Sketch and Scale Geo-distributed tSNE and UMAP

Abstract

Running machine learning analytics over geographically distributed datasets is a rapidly arising problem in the world of data management policies ensuring privacy and data security. Visualizing high dimensional data using tools such as t-distributed Stochastic Neighbor Embedding (tSNE) and Uniform Manifold Approximation and Projection (UMAP) became a common practice for data scientists. Both tools scale poorly in time and memory. While recent optimizations showed successful handling of 10,000 data points, scaling beyond million points is still challenging. We introduce a novel framework: Sketch and Scale (SnS). It leverages a Count Sketch data structure to compress the data on the edge nodes, aggregates the reduced size sketches on the master node, and runs vanilla tSNE or UMAP on the summary, representing the densest areas, extracted from the aggregated sketch.

Multi‐level refined power system operation mode analysis: A data‐driven approach

“…UMAP is a dimensionality reduction technique based on Riemannian geometry and algebraic topology theory used for non‐linear dimensionality reduction [29]. It has high speed when processing large data sets and can effectively retain the global structure of the data [31].…”