Optimizing graph layout by t-SNE perplexity estimation

Abstract

Perplexity is one of the key parameters of dimensionality reduction algorithm of t-distributed stochastic neighbor embedding (t-SNE). In this paper, we investigated the relationship of t-SNE perplexity and graph layout evaluation metrics including graph stress, preserved neighborhood information and visual inspection. As we found that a small perplexity is correlated with a relative higher normalized stress while preserving neighborhood information with a higher precision but less global structure information, we proposed our method to estimate appropriate perplexity either based on a modified standard t-SNE or the sklearn Barnes–Hut TSNE. Experimental results demonstrate effectiveness and ease of use of our approach when tested on a set of benchmark datasets.

Machine Learning for Enhanced Classroom Homogeneity in Primary Education

“…Multidimensional datasets often become sparse as the number of features increases, which can negatively impact clustering success. To address this, increasing the sample size or using dimensionality reduction algorithms like Principal Component Analysis, Linear Discriminant Analysis, Factor Analysis, t-SNE, and others can help reduce dimensions and improve data density (Gao et al, 2021;Wang et al, 2021;Mair, 2018;Xiao et al, 2023;Groth et al, 2013). Before creating a hypothesis, it is critical to evaluate the impact of each feature on cluster results using methods such as information retrieval.…”

“…When the data size is too large, using principal component analysis (PCA) or t-distributed stochastic neighbor embedding (Xiao et al, 2023) can first reduce the size of the data and then apply DBSCAN.…”