Diluvian Clustering: A Fast, Effective Algorithm for Clustering Compositional and Other Data

Abstract

Diluvian Clustering is an unsupervised grid-based clustering algorithm well suited to interpreting large sets of noisy compositional data. The algorithm is notable for its ability to identify clusters that are either compact or diffuse and clusters that have either a large number or a small number of members. Diluvian Clustering is fundamentally different from most algorithms previously applied to cluster compositional data in that its implementation does not depend upon a metric. The algorithm reduces in two-dimensions to a case for which there is an intuitive, real-world parallel. Furthermore, the algorithm has few tunable parameters and these parameters have intuitive interpretations. By eliminating the dependence on an explicit metric, it is possible to derive reasonable clusters with disparate variances like those in real-world compositional data sets. The algorithm is computationally efficient. While the worst case scales as O(N²) most cases are closer to O(N) where N is the number of discrete data points. On a mid-range 2014 vintage computer, a typical 20,000 particle, 30 element data set can be clustered in a fraction of a second.

Reproducible Spectrum and Hyperspectrum Data Analysis Using NeXL

“…Some of the more successful include Sandia's AXSIA (software application) (Kotula et al, 2003) (commercially available as Thermo Scientific's COMPASS tool) and the expectation-maximization algorithm (Dempster et al, 1977), hierarchical clustering tools (Guess & Wilson, 2002), nonhierarchical clustering tools like k -means (MacQueen et al, 1967), and supervised learning models like support vector machines (Cortes & Vapnik, 1995). Diluvian Clustering (Ritchie, 2015) was proposed as a simple yet effective mechanism for clustering k -ratio data from particles. This is the first documented application of the algorithm to X-ray spectrum image data.…”

Chemical Compound Classification by Elemental Signatures in Castle Dust Using SEM Automated X-ray Particle Analysis

“…The sum of the 4 SDD spectra (dwell time of 400 ms per spectrum) were used for quantification using NIST Graf [6]. A novel algorithm was then used to cluster the data obtained for 39,491 particles [7]. Data were then reprocessed using a manually developed rule set.…”