Scalable streaming tools for analyzing N-body simulations: Finding halos and investigating excursion sets in one pass

Abstract

Cosmological N-body simulations play a vital role in studying models for the evolution of the Universe. To compare to observations and make a scientific inference, statistic analysis on large simulation datasets, e.g., finding halos, obtaining multi-point correlation functions, is crucial. However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper [22] proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to 10 9 particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets. This paper presents a robust streaming tool that leverages state-of-the-art techniques on GPU boosting, sampling, and parallel I/O, to significantly improve performance and scalability. Our rigorous analysis of the sketch parameters improves the previous results from finding the centers of the 10 3 largest halos [22] to ∼ 10 4 − 10 5 , and reveals the trade-offs between memory, running time and number of halos. Our experiments show that our tool can scale to datasets with up to ∼ 10 12 particles while using less than an hour of running time on a single GPU Nvidia GTX 1080.

Large-scale dark matter simulations

“…The somewhat arbitrary definition of a halo and its boundary has led to a multitude of operational definitions in N-body simulations and thus numerous algorithms exist (Davis et al 1985;Eisenstein and Hut 1998;Stadel 2001;Bullock et al 2001;Springel et al 2001a;Aubert et al 2004;Gill et al 2004;Weller et al 2005;Neyrinck et al 2005;Kim and Park 2006;Diemand et al 2006;Shaw et al 2007;Maciejewski et al 2009;Knollmann and Knebe 2009;Planelles and Quilis 2010;Habib et al 2009;Behroozi et al 2013b;Skory et al 2010;Falck et al 2012;Roy et al 2014;Ivkin et al 2018;Elahi et al 2019); these methods can be roughly split into percolation algorithms in configuration or phase-space, and identification of peaks in the density field. Knebe et al (2011Knebe et al ( , 2013 performed a systematic comparison among 17 halo finders based on the same mock halos and N-body simulation.…”

Mining Discriminative K-Mers in DNA Sequences Using Sketches and Hardware Acceleration

“…Preliminary results indicate that a CM-CU sketch achieves accurate frequency estimates with low memory requirements compared to CM and CS, thus enabling hardware accelerators that leverage on-chip memory resources to increase performance [33]. Beyond FPGA-based systems, authors in [35] use GPUs to accelerate halo-finding in cosmological N-body simulation data using sketches.…”

Sketch and Scale Geo-distributed tSNE and UMAP

“…3) Streaming sketches of heavy hitters: The field of streaming algorithms arose from the necessity to process massive data in sequential order while operating in a very low memory. First introduced for the estimation of the frequency moments [1], it was further developed for a wide spectrum of problems within linear algebra [28], graph problems [14], and others [16]; and found applications in machine learning [9], [20], networking [7], [11], astrophysics [8], [10]. Further we provide a glimpse on streaming model and sketches for finding frequent items.…”

“…We find 10 clusters in the data, labeled from 1 through 10, as shown on Figure 3. These can be grouped into three categories, pixels related to tumor (1,5,8,10), pixel related to nuclei of cells (6,9), and non-tumor tissue (2,3,4,7). These show an excellent agreement with labels generated for nuclei using an industry standard segmentation software.…”