A systematic performance evaluation of clustering methods for single-cell RNA-seq data

Abstract

Subpopulation identification, usually via some form of unsupervised clustering, is a fundamental step in the analysis of many single-cell RNA-seq data sets. This has motivated the development and application of a broad range of clustering methods, based on various underlying algorithms. Here, we provide a systematic and extensible performance evaluation of 14 clustering algorithms implemented in R, including both methods developed explicitly for scRNA-seq data and more general-purpose methods. The methods were evaluated using nine publicly available scRNA-seq data sets as well as three simulations with varying degree of cluster separability. The same feature selection approaches were used for all methods, allowing us to focus on the investigation of the performance of the clustering algorithms themselves. We evaluated the ability of recovering known subpopulations, the stability and the run time and scalability of the methods. Additionally, we investigated whether the performance could be improved by generating consensus partitions from multiple individual clustering methods. We found substantial differences in the performance, run time and stability between the methods, with SC3 and Seurat showing the most favorable results. Additionally, we found that consensus clustering typically did not improve the performance compared to the best of the combined methods, but that several of the top-performing methods already perform some type of consensus clustering. All the code used for the evaluation is available on GitHub ( https://github.com/markrobinsonuzh/scRNAseq_clustering_comparison). In addition, an R package providing access to data and clustering results, thereby facilitating inclusion of new methods and data sets, is available from Bioconductor ( https://bioconductor.org/packages/DuoClustering2018).

Current best practices in single‐cell RNA‐seq analysis: a tutorial

“…This method is the default clustering method implemented in the Scanpy and Seurat single-cell analysis platforms. It has been shown to outperform other clustering methods for single-cell RNAseq data (Duò et al, 2018;Freytag et al, 2018), and flow and mass cytometry data (Weber & Robinson, 2016). Conceptually, the Louvain algorithm detects communities as groups of cells that have more links between them than expected from the number of links the cells have in total.…”

SAME-clustering: Single-cell Aggregated Clustering via Mixture Model Ensemble

“…The worst rank of 5.5 came from the Biase dataset, where only one cell was misplaced by SAME, leading to a high ARI but didn't perform well rank-wise because there were 3 methods that achieved perfect clustering when compared to the "gold-standard". We also compare our results to our previously published SAFE method (12), which performed overall second best and remains an attractive alternative (33), particularly when analyzing large datasets to save computation time.…”

Straightforward clustering of single-cell RNA-Seq data with t-SNE and DBSCAN

“…In their systematic evaluation of clustering methods, Duò et al included a method in which cells are clustered by applying the k-means algorithm to t-SNE results. However, kmeans is not a suitable choice for clustering data after applying a non-linear dimensionality reduction method such as t-SNE that only preserves local, but not global distances 5 . It is sometimes believed that t-SNE results have merely "illustrative" character and are not sufficiently reliable to justify their use in clustering.…”

“…The unique characteristics of single-cell RNA-Seq data, in particular the high intrinsic levels of technical noise [2][3][4] , have motivated the development of specialized scRNA-Seq clustering methods. A wide array of methods has been proposed, and a recent study has systematically compared twelve clustering methods on real and simulated data 5 .…”