CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data

Abstract

Mapping biological mechanisms in cellular systems is a fundamental step in earlystage drug discovery that serves to generate hypotheses on what disease-relevant molecular targets may effectively be modulated by pharmacological interventions. With the advent of high-throughput methods for measuring single-cell gene expression under genetic perturbations, we now have effective means for generating evidence for causal gene-gene interactions at scale. However, inferring graphical networks of the size typically encountered in real-world gene-gene interaction networks is difficult in terms of both achieving and evaluating faithfulness to the true underlying causal graph. Moreover, standardised benchmarks for comparing methods for causal discovery in perturbational single-cell data do not yet exist. Here, we introduce CausalBench -a comprehensive benchmark suite for evaluating network inference methods on large-scale perturbational single-cell gene expression data. CausalBench introduces several biologically meaningful performance metrics and operates on two large, curated and openly available benchmark data sets for evaluating methods on the inference of gene regulatory networks from single-cell data generated under perturbations. With real-world datasets consisting of over 200 000 training samples under interventions, CausalBench could potentially help facilitate advances in causal network inference by providing what is -to the best of our knowledge -the largest openly available test bed for causal discovery from real-world perturbation data to date.

Gene regulatory network structure informs the distribution of perturbation effects

“…Several computational and experimental approaches have been used to estimate pairwise causal relationships between genes, with the ultimate goal of wholesale inference of gene regulatory networks [10,11]. These data and methods are broad in scope, and range from estimating networks using natural variation in gene expression values from bulk tissue [12,23] to fitting complex machine learning models on data from single-cell perturbation experiments [7,15,32].…”

“…But neither form of data are a panacea, and care is warranted in the analysis of experimental data and in the development of structure learning algorithms. For example, sorting and thresholding perturbation effects has been shown to be a high-quality baseline for network reconstruction [10,11] (one that we also use to compare networks in Fig. 5).…”

“…Meanwhile, recent advances in the design of interventional studies, including CRISPR-based molecular perturbation approaches like Perturb-seq [4,5], have been useful for learning the local structure of GRNs around a focal gene or pathway [6,7], discovering trait-relevant gene sets at scale [8], and determining novel functions for poorly characterized genes in a particular cell type [9]. The preponderance of single-cell data in multiple cell types, tissues, and contexts has also fueled a resurgence of interest in the wholesale inference of GRNs, capitalizing on new techniques from graph theory and causal inference [10,11].…”

Higher-Order Least Squares: Assessing Partial Goodness of Fit of Linear Causal Models

“…We consider the K562 dataset provided by Replogle et al (2022). We follow the preprocessing in the benchmark of Chevalley et al (2022). Then, the dataset contains 162,751 measurements of the activity of 622 genes: 10,691 of the measurements are taken in a purely observational environment while the remaining are obtained under various interventions.…”

Stochastic Modeling of Biophysical Responses to Perturbation

“…The goal of such work is to unravel how the processes of DNA/RNA regulation produce observed cellular responses and diversity. This provides fertile ground for mechanistic studies of transcription, to tease apart how features such as DNA structure, genetic interactions, and mRNA processing dynamics [6][7][8] regulate processes like cellular differentiation and disease progression. For example, in many published single-cell RNA sequencing (scRNA-seq) datasets [9], one can obtain information about nascent and mature ('unspliced' and 'spliced') mRNA expression, which represents different components of mRNA production and processing [10].…”