Stochastic quartet approach for fast multidimensional scaling

Abstract

Multidimensional scaling is a statistical process that aims to embed high-dimensional data into a lower-dimensional, more manageable space. Common MDS algorithms tend to have some limitations when facing large data sets due to their high time and spatial complexities. This paper attempts to tackle the problem by using a stochastic approach to MDS which uses gradient descent to optimise a loss function defined on randomly designated quartets of points. This method mitigates the quadratic memory usage by computing distances on the fly, and has iterations in O(N ) time complexity, with N samples. Experiments show that the proposed method provides competitive results in reasonable time. Public codes are available at https://github.com/PierreLambert3/SQuaD-MDS.git. Multidimensional scaling and its limitationsDimensionality reduction (DR) is the process of mapping high-dimensional (HD) observations into a lower-dimensional (LD) space such that the LD embedding is a faithful representation of the HD data. The main DR uses are in machine learning, to curb the curse of dimensionality, and in visualisation. Mapped data can reveal structures that would lay hidden from the human perception if left in HD. Typically, some information is lost by the DR and, therefore, each DR method has a take on what kind of information should be preserved and what can be lost. Used frequently in visualisation, t-SNE [1] aims at retaining the neighbourhood of each point according to a distance metric and a perplexity, which reflects the size of the neighbourhood to preserve. While t-SNE excels at retaining local structures, sufficiently remote points tend to be considered equally distant by the algorithm and, therefore, the larger-scale structures can be distorted. Such distortions can lead to erroneous conclusions by the human user, who might overestimate the dissimilarity between two clusters that are distant in the LD embedding. For this reason, using multiple DR paradigms in conjunction is a good practice in visualisation: another embedding that preserves distances instead of neighbourhoods would have prevented this erroneous conclusion.This paper considers metric multidimensional scaling (MDS): a DR technique that produces a LD embedding such that the pairwise distances in LD reflect those in HD. MDS minimises a cost function which, in its simplest form, is the sum of the squared differences between distances in HD and the Euclidean distances in LD. A common strategy to optimize this cost function is based on 417

Interpretable models for scRNA-seq data embedding with multi-scale structure preservation

“…C . A dimension-reduction model based on a variational autoencoder (VAE) architecture and quartet loss regularisation, adapted from SQuadMDS (26), is trained using denoised data. Normalised distances within randomly drawn quartets of points are optimised jointly, so as to impose a multi-scale structure-preservation constraint on the latent space.…”

“…ViVAE denoises high-dimensional input data and learns a lower-dimensional representation using a variational autoencoder (VAE) architecture, while imposing a structure-preserving constraint to optimise both local and global distances between points. This constraint, the quartet loss , is an adaptation of the main idea in SQuadMDS (26). It seeks to preserve relative distances within quartets (groups of 4) of points that are randomly (repeatedly) drawn out of the input dataset and the lower-dimensional latent embedding of it.…”

“…In our comparative analysis, we apply multiple existing DR algorithms as well as our newly proposed one. These are principal component analysis (PCA), t -SNE (35), UMAP (37), PaCMAP (57), TriMap (1), SQuadMDS (26), ivis (50) and ViVAE . We test all methods on a collection of large scRNA-seq (48; 16; 63; 43; 67) and mass cytometry (CyTOF) (31; 46) datasets to evaluate SP in their respective low-dimensional embeddings.…”

“…More recently proposed algorithms ( TriMap (1), SQuadMDS (26), fmSNE (6)) set out to preserve global relationships better or to strike a new balance between local and global structure preservation ( ivis (50), PaCMAP (57)). We welcome this trend, due to the promise of reducing misinterpretation of t -SNE and UMAP as two popular, locally focused methods that are used heavily to visualise data with important hierarchical or global structures.…”

SQuadMDS: a lean Stochastic Quartet MDS improving global structure preservation in neighbor embedding like t-SNE and UMAP

“…To keep the computational complexity of each SGD iteration low, this works takes a divide-and-conquer approach to compute the gradients. At each iteration, the data set is randomly partitioned into groups of 4 points called quartets [19]. The previously stated mMDS stress function is redefined to put independent quartets of points to the forefront, in order to compute the gradients for each point by looking only at the three other points in the quartet at each SGD iteration.…”