Fast algorithms for comprehensive n-point correlation estimates

Abstract

The n-point correlation functions (npcf) are powerful spatial statistics capable of fully characterizing any set of multidimensional points. These functions are critical in key data analyses in astronomy and materials science, among other fields, for example to test whether two point sets come from the same distribution and to validate physical models and theories. For example, the npcf has been used to study the phenomenon of dark energy, considered one of the major breakthroughs in recent scientific discoveries. Unfortunately, directly estimating the continuous npcf at a single value requires O(N n ) time for N points, and n may be 2, 3, 4 or even higher, depending on the sensitivity required. In order to draw useful conclusions about real scientific problems, we must repeat this expensive computation both for many different scales in order to derive a smooth estimate and over many different subsamples of our data in order to bound the variance.We present the first comprehensive approach to the entire n-point correlation function estimation problem, including fast algorithms for the computation at multiple scales and for many subsamples. We extend the current state-of-the-art tree-based approach with these two algorithms. We show an order-of-magnitude speedup over the current best approach with each of our new algorithms and show that they can be used together to obtain over 500x speedups over the stateof-the-art in order to enable much larger datasets and more accurate scientific analyses than were possible previously.

Dual‐tree fast exact max‐kernel search

“…This is a huge improvement over the linear scan all-nearest-neighbors runtime of O(N 2 ). Similar runtime bound results have been shown for some dualtree algorithms when cover trees are chosen as the tree type [58,45,46].…”

“…More recently, Gray and Moore proposed using a second tree for problems with large query sets [25], such as all-nearest-neighbors and density estimation [26]. This dual-tree approach was then applied to numerous problems: singular value decomposition [28], n-point correlation estimates in astronomy [45], mean shift [63], kernel summation [39,41,40], rankapproximate nearest neighbor search [58], and minimum spanning tree calculation [46], as well as numerous others.…”

Plug-and-play dual-tree algorithm runtime analysis

“…There exist numerous dual-tree algorithms for problems as diverse as kernel density estimation (Gray and Moore, 2003), mean shift (Wang et al, 2007), minimum spanning tree calculation (March et al, 2010), n-point correlation function estimation (March et al, 2012), max-kernel search (Curtin et al, 2013c), particle smoothing (Klaas et al, 2006), variational inference (Amizadeh et al, 2012), range search (Gray and Moore, 2001), and embedding techniques Van Der Maaten (2014), to name a few.…”

Tree-Independent Dual-Tree Algorithms

“…tion, important in astrophysics, is an n-body problem and can be solved quickly with trees (March et al, 2012). In addition, Euclidean minimum spanning trees can be found quickly using tree-based algorithms (March et al, 2010).…”