Streaming Algorithms for Halo Finders

Liu, Zaoxing; Ivkin, Nikita; Yang, Lin F.; Neyrinck, Mark C.; Lemson, Gerard; Szalay, Alexander S.; Braverman, Vladimir; Budavàri, Tamás; Burns, Randal; Wang, Xin

doi:10.1109/escience.2015.73

Cited by 4 publications

(15 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our tool needs less than 5 minutes to find the top 3 · 10 5 heavy cells on a dataset with 10 10 particles. Compared to previous results [22], which required more than 8 hours, it's more than a 100× improvement. This dataset consists of a snapshot of the Millennium dataset [35] and we use a grid of 10 11 cells in our algorithm for approximation of the density field, which can be used further for astrophysical analysis.…”

Section: Introductionmentioning

confidence: 62%

“…Thus for state-of-the-art simulations [2,32], which reach hundreds of billions and even over a trillion of particles, post-processing analysis becomes unfeasible unless using supercomputers of the same size that created the simulations in the first place. Recently in [22] an approach was proposed that attacks the problem using solutions developed in the field of streaming algorithms. Typical applications of streaming algorithms are very large datasets, where access to the data is restricted to be sequential and the working memory is much smaller than the dataset size.…”

Section: Introductionmentioning

confidence: 99%

“…In [22], we developed a solution using the Count Sketch streaming algorithm [6] to the halo finding problem, and presented the first results on how to find the top-k (k ≈ 1000) largest halos in a dark matter simulation. In that paper, all experiments were running on relatively small data streams with at most 10 9 items.…”

Section: Introductionmentioning

confidence: 99%

“…In [22] authors reduced the halo-finding problem to the problem of finding the top-k densest cells in a regular mesh. This reduction shows that these densest cells are closely related to the space with the heaviest halos.…”

Section: Introductionmentioning

confidence: 99%

“…However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper [22] proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to 10 9 particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Scalable streaming tools for analyzing N-body simulations: Finding halos and investigating excursion sets in one pass

Ivkin

Liu

Yang

et al. 2018

Astronomy and Computing

View full text Add to dashboard Cite

Cosmological N-body simulations play a vital role in studying models for the evolution of the Universe. To compare to observations and make a scientific inference, statistic analysis on large simulation datasets, e.g., finding halos, obtaining multi-point correlation functions, is crucial. However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper [22] proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to 10 9 particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets. This paper presents a robust streaming tool that leverages state-of-the-art techniques on GPU boosting, sampling, and parallel I/O, to significantly improve performance and scalability. Our rigorous analysis of the sketch parameters improves the previous results from finding the centers of the 10 3 largest halos [22] to ∼ 10 4 − 10 5 , and reveals the trade-offs between memory, running time and number of halos. Our experiments show that our tool can scale to datasets with up to ∼ 10 12 particles while using less than an hour of running time on a single GPU Nvidia GTX 1080.

show abstract

Section: Introductionmentioning

confidence: 62%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Scalable streaming tools for analyzing N-body simulations: Finding halos and investigating excursion sets in one pass

Ivkin

Liu

Yang

et al. 2018

Astronomy and Computing

View full text Add to dashboard Cite

show abstract

Sketch and Scale Geo-distributed tSNE and UMAP

Wei

Ivkin

Braverman

et al. 2020

2020 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Running machine learning analytics over geographically distributed datasets is a rapidly arising problem in the world of data management policies ensuring privacy and data security. Visualizing high dimensional data using tools such as t-distributed Stochastic Neighbor Embedding (tSNE) and Uniform Manifold Approximation and Projection (UMAP) became a common practice for data scientists. Both tools scale poorly in time and memory. While recent optimizations showed successful handling of 10,000 data points, scaling beyond million points is still challenging. We introduce a novel framework: Sketch and Scale (SnS). It leverages a Count Sketch data structure to compress the data on the edge nodes, aggregates the reduced size sketches on the master node, and runs vanilla tSNE or UMAP on the summary, representing the densest areas, extracted from the aggregated sketch.

show abstract

Joltik

Yang

Zhang

Gadre

et al. 2020

Proceedings of the 26th Annual International Conference on Mobile Computing and Networking

View full text Add to dashboard Cite

Wireless sensors have enabled a number of key applications. Due to their energy constraints, wireless sensors today communicate occasional short samples or predetermined summary statistics of the data they collect. This means that computing every additional statistic at high fidelity incurs additional communication and energy overhead. This paper presents Joltik, a framework enabling general, future-proof, and energy-efficient analytics for low power wireless sensors. Joltik is general in that it summarizes sensed data from low-power devices without making assumptions on which specific statistical metric(s) are desired at the cloud and is future-proof, meaning it supports new, unforeseen metrics. Joltik is built upon recent theoretical advances in universal sketching, which can enable a Joltik sensor node to report a compact summary of observed data to enable a large class of statistical summaries. We address key system design and implementation challenges with respect to communication, memory, and computation bottlenecks that arise in practically realizing the potential benefits of universal sketching in the low-power regime. We present a proof-of-concept testbed evaluation of Joltik in LoRaWAN NUCLEO-L476RG boards and sensors. Across a range of realistic datasets, Joltik provides up to a 24.6× reduction in energy cost compared to transmitting raw data and outperforms many natural alternatives (e.g., sub-sampling, custom sketches, compressed sensing, and lossy compression) in terms of energy-accuracy trade-offs. CCS Concepts • Networks → Network components; • Computer systems organization → Embedded and cyber-physical systems; • Hardware → Communication hardware, interfaces and storage.

show abstract

Streaming Algorithms for Halo Finders

Cited by 4 publications

References 29 publications

Scalable streaming tools for analyzing N-body simulations: Finding halos and investigating excursion sets in one pass

Scalable streaming tools for analyzing N-body simulations: Finding halos and investigating excursion sets in one pass

Sketch and Scale Geo-distributed tSNE and UMAP

Joltik

Contact Info

Product

Resources

About