GigaSOM.jl: High-performance clustering and visualization of huge cytometry datasets

Abstract

Background: The amount of data generated in large clinical and phenotyping studies that use single-cell cytometry is constantly growing. Recent technological advances allow to easily generate data with hundreds of millions of single-cell data points with more than 40 parameters, originating from thousands of individual samples. The analysis of that amount of high-dimensional data becomes demanding in both hardware and software of high-performance computational resources. Current software tools often do not scale to the datasets of such size; users are thus forced to downsample the data to bearable sizes, in turn losing accuracy and ability to detect many underlying complex phenomena. Results: We present GigaSOM.jl, a fast and scalable implementation of clustering and dimensionality-reduction for flow and mass cytometry data. The implementation of GigaSOM.jl in the high-level and high-performance programming language Julia makes it accessible to the scientific community, and allows for efficient handling and processing of datasets with billions of data points using distributed computing infrastructures. We describe the design of GigaSOM.jl, measure its performance and horizontal scaling capability, and showcase the functionality on a large dataset from a recent study. Conclusions: GigaSOM.jl facilitates utilization of the commonly available high-performance computing resources to process the largest available datasets within minutes, while producing results of the same quality as the current state-of-art software. Measurements indicate that the performance scales to much larger datasets. The example use on the data from an massive mouse phenotyping effort confirms the applicability of GigaSOM.jl to huge-scale studies.

Scalable semi-supervised dimensionality reduction with GPU-accelerated EmbedSOM

“…The ability to e ectively work with a simpli ed model of the data di erentiates it from other dimensionality reduction methods; in turn it o ers superior performance by reducing the amount of necessary computation as well as by opening parallelization potential, since the computations of the projections of many individual points are independent. In the setting of ow and mass cytometry data visualization, this provided speedup of several orders of magnitude against the other available methods [14,16].…”

“…EmbedSOM provided an order-of-magnitude speedup on datasets typical for the single-cell cytometry data visualization while retaining competitive quality of the results. The concept has proven useful for interactive and high-performance work ows in cytometry [16,14], and easily applies to many other types of datasets. Despite of that, the parallelization potential of the extremely data-regular design of EmbedSOM algorithm has remained mostly untapped.…”

Animal Movement Strategies

“…The potential stumbling block here is that the methods required for such clustering are also not native to ecology and evolution, and themselves suffer from being much too complex to interpret for a general biologist audience (see e.g. the GigaSOM method for clustering single-cell cytometry data; where SOM is a 'self organised map', a form of machine learning Kratochvíl et al 2020). Furthermore, it is also unclear how these mechanisms should undergo evolution -in , we suggested using both genetic algorithms and reinforcement learning acting on the simulated mechanisms, based on the similarity of simulated movement paths with real animal movements.…”