The shaky foundations of simulating single-cell RNA sequencing data

Abstract

Background With the emergence of hundreds of single-cell RNA-sequencing (scRNA-seq) datasets, the number of computational tools to analyze aspects of the generated data has grown rapidly. As a result, there is a recurring need to demonstrate whether newly developed methods are truly performant—on their own as well as in comparison to existing tools. Benchmark studies aim to consolidate the space of available methods for a given task and often use simulated data that provide a ground truth for evaluations, thus demanding a high quality standard results credible and transferable to real data. Results Here, we evaluated methods for synthetic scRNA-seq data generation in their ability to mimic experimental data. Besides comparing gene- and cell-level quality control summaries in both one- and two-dimensional settings, we further quantified these at the batch- and cluster-level. Secondly, we investigate the effect of simulators on clustering and batch correction method comparisons, and, thirdly, which and to what extent quality control summaries can capture reference-simulation similarity. Conclusions Our results suggest that most simulators are unable to accommodate complex designs without introducing artificial effects, they yield over-optimistic performance of integration and potentially unreliable ranking of clustering methods, and it is generally unknown which summaries are important to ensure effective simulation-based method comparisons.

A comparison of marker gene selection methods for single-cell RNA sequencing data

“…We use the splat simulation model implemented in the splatter R package to simulate scRNA-seq datasets [ 48 ]. The splat model is the most widely used simulation model in scRNA-seq benchmarking studies [ 49 ] and was recently shown to be in the top tier of simulation methods [ 50 ]. To simulate marker genes specifically, we designed and implemented a novel marker gene score to select and rank marker genes using the splat model’s cluster-specific differential expression parameters ( Methods ).…”

“…A limitation of the splat model is that it does not allow these cluster-specific differential expression parameters to be estimated from data. Indeed, no current simulation scRNA-seq method incorporates this facility [ 49 ]. To overcome this limitation in practice we performed analyses comparing simulated and true marker genes to identify values of the parameters that were able to recapitulate true expert-annotated marker genes (Additional file 1 : Figs.…”

Imbalance and Composition Correction Ensemble Learning Framework (ICCELF): A novel framework for automated scRNA-seq cell type annotation

“…The simulated datasets were generated using the SPARsim [7] package in R [8], which creates count matrices resembling real data by modeling the distribution of zeros with a Gamma-Multivariate hypergeometric distribution. In evaluations by Crowell et al [9] and Cao et al [10], SPARsim emerged as one of the top performing scRNA-seq simulators, closely mimicking real data properties. To set the SPARsim parameters, we estimated them from the 10X Genomics example datasets of human Jurkat and 293T cells from Zheng et al The 293T sample of 1,718 cells is included as a built-in SPARsim dataset.…”

Systematic comparison of sequencing-based spatial transcriptomic methods

“…However, a systematic benchmarking study has not been done for sST. Prior studies have established frameworks for comparing single-cell transcriptomic and epigenomic methods, underscoring the necessity for standardized evaluation criteria and reference tissues for technology validation [6–9], since simulated single-cell and spatial data may not be reliable [10]. While sST technologies share common features, such as the use of spatial DNA barcodes analogous to cell barcodes in scRNA-seq, the methods diverge significantly in aspects like spatial resolution and the preparation of spatially barcoded oligo arrays [11].…”