Identifying strengths and weaknesses of methods for computational network inference from single cell RNA-seq data

Abstract

Single-cell RNA-sequencing (scRNA-seq) offers unparalleled insight into the transcriptional pro- grams of different cellular states by measuring the transcriptome of thousands individual cells. An emerging problem in the analysis of scRNA-seq is the inference of transcriptional gene regulatory net- works and a number of methods with different learning frameworks have been developed. Here we present a expanded benchmarking study of eleven recent network inference methods on six published single-cell RNA-sequencing datasets in human, mouse, and yeast considering different types of gold standard networks and evaluation metrics. We evaluate methods based on their computing requirements as well as on their ability to recover the network structure. We find that while no method is a universal winner and most methods have a modest recovery of experimentally derived interactions based on global metrics such as AUPR, methods are able to capture targets of regulators that are relevant to the system under study. Based on overall performance we grouped the methods into three main categories and found a combination of information-theoretic and regression-based methods to have a generally high perfor- mance. We also evaluate the utility of imputation for gene regulatory network inference and find that a small number of methods benefit from imputation, which further depends upon the dataset. Finally, comparisons to inferred networks for comparable bulk conditions showed that networks inferred from scRNA-seq datasets are often better or at par to those from bulk suggesting that scRNA-seq datasets can be a cost-effective way for gene regulatory network inference. Our analysis should be beneficial in selecting algorithms for performing network inference but also argues for improved methods and better gold standards for accurate assessment of regulatory network inference methods for mammalian systems.

Network inference with Granger causality ensembles on single-cell transcriptomics

“…Inferring GRNs from single-cell gene expression data remains a difficult task. Evaluations of network inference algorithms on simulated (Chen and Mar, 2018) and real (Stone et al, 2021) single-cell datasets reported that predictions were generally only slightly better than random edge ordering. New single-cell gene expression simulators designed specifically for simulating GRNs (Pratapa et al, 2020;Dibaeinia and Sinha, 2020) can help inform which GRN inference methods are best for different types of biological trajectories.…”

GRaNIE and GRaNPA: Inference and evaluation of enhancer-mediated gene regulatory networks applied to study macrophages

“…An important step in the regulatory network reconstruction is to evaluate their biological significance. Common approaches for assessing regulatory interactions include benchmarking them against networks from simulated data or against known biological networks [22][23][24]. The drawback of simulated networks is that they are based on many assumptions of the structure of a "true" biological network.…”

“…Benchmarking against true biological networks typically suffer from a strong literature bias [25], low complexity of these “true” circuits and a limited range of connections and cell types. In general, each network inference method will have its own bias and shortcomings, and performance will depend on the benchmarking dataset [22–24]. Thus, there is a strong need for an unbiased approach to assess the biological relevance of inferred regulatory interactions.…”

“…GRaNPA is a general framework for evaluating the biological relevance of TF-gene GRNs by assessing their performance for predicting cell-type specific differential expression. We demonstrate the power of our tool-suite by investigating gene regulatory mechanisms in macrophages that underlie their response to infection, and their involvement in common genetic diseases including autoimmune diseases.the benchmarking dataset [22][23][24]. Thus, there is a strong need for an unbiased approach to assess the biological relevance of inferred regulatory interactions.Here we present a tool-suite for building and evaluating enhancer-inclusive gene regulatory networks (eGRNs) called GRaNIE (Gene Regulatory Network Inference including Enhancers -https://git.embl.de/grp-zaugg/GRaNIE) and GRaNPA (Gene Regulatory Network Performance Analysis -https://git.embl.de/grp-zaugg/GRaNPA), respectively.…”

“…the benchmarking dataset [22][23][24]. Thus, there is a strong need for an unbiased approach to assess the biological relevance of inferred regulatory interactions.…”