A Systematic Evaluation of Single-cell RNA-sequencing Imputation Methods

Abstract

6The rapid development of single-cell RNA-sequencing (scRNA-seq) technology, with increased sparsity compared to bulk RNA-sequencing (RNA-seq), has led to the emergence of many methods for preprocessing, including imputation methods. Here, we systematically evaluate the performance of 18 state-of-the-art scRNA-seq imputation methods using cell line and tissue data measured across experimental protocols. Specifically, we assess the similarity of imputed cell profiles to bulk samples as well as investigate whether methods recover relevant biological signals or introduce spurious noise in three downstream analyses: differential expression, unsupervised clustering, and inferring pseudotemporal trajectories. Broadly, we found significant variability in the performance of the methods across evaluation settings. While most scRNA-seq imputation methods recover biological expression observed in bulk RNA-seq data, the majority of the methods do not improve performance in downstream analyses compared to no imputation, in particular for clustering and trajectory analysis, and thus should be used with caution. Furthermore, we find that the performance of scRNA-seq imputation methods depends on many factors including the experimental protocol, the sparsity of the data, the number of cells in the dataset, and the magnitude of the effect sizes. We summarize our results and provide a key set of recommendations for users and investigators to navigate the current space of scRNA-seq imputation methods. 7 Figure 1. Motivation and overview of benchmark evaluation of scRNA-seq imputation methods. (A) Dimension reduction results after applying Principal Components Analysis (PCA) from either no imputation method (no_imp highlighted in red) or the 18 imputation methods using the null simulations data (Section 5.3) in which no structural pattern is expected. The color represents the simulated library size (defined as the total sum of counts across all relevant features) for each cell. (B) An overview of the benchmark comparison evaluating 18 scRNA-seq imputation methods.with model-based methods outperforming smoothing-based methods, in particular for genes with a small effect size (log2 1 fold-change) 14 . Another study found imputation methods can introduce spurious correlations between imputed expression 2 and total UMI counts 15 . Alternatively, others have shown spurious structural patterns in low dimensional representations of 3 imputed data 14, 18 , which we also find in data where we expect no structural patterns in the data but patterns associated with 4 library size emerge in the imputed data ( Figures 1A, S1). In contrast, others have found a subset of imputation methods to be 5 helpful to estimate library size factors for normalization of sparse scRNA-seq data 16 . Therefore, the answer to the question of 6 which methods can, let alone should, be used for a particular analysis is often unclear. 7To address this gap, we performed a systematic benchmark comparison and evaluation of 18 state-of-the-art scRNA-seq 8 imputation met...

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single cell RNA-seq preprocessing tools

“…A previous systematic comparison [18] instead found that the use of SAVERX for denoising data and scran for clustering yielded better performance on the ability to recover differentially expressed genes, whereas we focused on the ability to recover cell populations. Another disagreement with the present recommendations come from the study of Hou et al [10] that recommends SAVERX and discourage the use of DCA for better clustering, whereas our study found opposite results for these tools. Several differences in the design of our study could explain this contrast, such as the use of different metrics (ARI at true number of clusters, mean precision/ recall), of different datasets (Koh, Kumar, simMix simulations) and the use of different filtering and normalization strategies.…”

“…The performance of scVI in our evaluation was lower than in the original publication presenting the method [38]. Our results are however in line with other studies comparing silhouette score [58] and clustering accuracy [10] of scVI's latent space. These studies, like ours, used datasets with relatively few cells (i.e., fewer than genes), for which scVI was reported by its authors to be less adapted.…”

“…A number of good comparison and benchmark studies have already been performed on various steps related to scRNAseq processing and analysis and can guide the choice of methodology [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]. However, these recommendations need constant updating and often leave open many details of an analysis.…”

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single-cell RNA-seq preprocessing tools

“…A previous systematic compar- 411 ison [18] instead found that the use of SAVERX for denoising data and scran for clustering yielded 412 better performance on the ability to recover differentially-expressed genes, whereas we focused on 413 the ability to recover cell populations. Another disagreement with the present recommendations 414 come from the study of Hou et al [10] that recommends SAVERX and discourage the use of DCA 415 for better clustering, whereas our study found opposite results for these tools. Several differences 416 in the design of our study could explain this contrast, such as the use of different metrics (ARI 417 at true number of clusters, mean precision/ recall), of different datasets (Koh, Kumar, simMix 418 simulations) and the use of different filtering and normalization strategies.…”

“…The evaluation framework, pipeComp, has 8 been implemented so as to easily integrate any other step or tool, allowing extensible benchmarks 9 and easy application to other fields (https://github.com/plger/pipeComp). 10 Background 11 Single-cell RNA-sequencing (scRNAseq) and the set of attached analysis methods are evolving 12 fast, with more than 560 software tools available to the community [1] , roughly half of which are 13 dedicated to tasks related to data processing such as clustering, ordering, dimension reduction 14 or normalization. This increase in the number of available tools follows the development of new 15 sequencing technologies and the growing number of reported cells, genes and cell populations [2] .…”