Evaluating methods of inferring gene regulatory networks highlights their lack of performance for single cell gene expression data

Abstract

BackgroundA fundamental fact in biology states that genes do not operate in isolation, and yet, methods that infer regulatory networks for single cell gene expression data have been slow to emerge. With single cell sequencing methods now becoming accessible, general network inference algorithms that were initially developed for data collected from bulk samples may not be suitable for single cells. Meanwhile, although methods that are specific for single cell data are now emerging, whether they have improved performance over general methods is unknown. In this study, we evaluate the applicability of five general methods and three single cell methods for inferring gene regulatory networks from both experimental single cell gene expression data and in silico simulated data.ResultsStandard evaluation metrics using ROC curves and Precision-Recall curves against reference sets sourced from the literature demonstrated that most of the methods performed poorly when they were applied to either experimental single cell data, or simulated single cell data, which demonstrates their lack of performance for this task. Using default settings, network methods were applied to the same datasets. Comparisons of the learned networks highlighted the uniqueness of some predicted edges for each method. The fact that different methods infer networks that vary substantially reflects the underlying mathematical rationale and assumptions that distinguish network methods from each other.ConclusionsThis study provides a comprehensive evaluation of network modeling algorithms applied to experimental single cell gene expression data and in silico simulated datasets where the network structure is known. Comparisons demonstrate that most of these assessed network methods are not able to predict network structures from single cell expression data accurately, even if they are specifically developed for single cell methods. Also, single cell methods, which usually depend on more elaborative algorithms, in general have less similarity to each other in the sets of edges detected. The results from this study emphasize the importance for developing more accurate optimized network modeling methods that are compatible for single cell data. Newly-developed single cell methods may uniquely capture particular features of potential gene-gene relationships, and caution should be taken when we interpret these results.Electronic supplementary materialThe online version of this article (10.1186/s12859-018-2217-z) contains supplementary material, which is available to authorized users.

Current best practices in single‐cell RNA‐seq analysis: a tutorial

“…Uncovering these regulatory interactions is the goal of gene regulatory network (GRN) inference methods. Gene regulatory network inference is performed based on measurements of gene co-expression such as correlation, mutual information, or via regression models (Chen & Mar, 2018). If two genes show a co-expression signal even when all other genes are taken into account as potential confounders, these genes are said to have a causal regulatory relationship.…”

“…While there exist GRN inference methods that were specifically developed for scRNA‐seq data (SCONE: Matsumoto et al , ; PIDC: Chan et al , ; SCENIC: Aibar et al , ), a recent comparison has shown both bulk and single‐cell methods to perform poorly on these data (Chen & Mar, ). GRN inference methods may still offer valuable insights to identify causal regulators of biological processes, yet we recommend that these methods be used with care.…”

Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data

“…Next, we considered how to simulate these networks to create in silico single-cell gene expression datasets. Several recent studies on GRN inference from such data [7,9,10,18,27] have used GeneNetWeaver [23], a method originally developed for generating time courses of bulk-RNA datasets from a given GRN. Accordingly, we simulated the six synthetic networks using GeneNetWeaver via the procedure outlined in Chan et al [7] (see Supplementary Section S1 for details).…”

“…We start this section by giving an overview of GeneNetWeaver [23], a popular method for simulating bulk gene expression datasets. It is being used increasingly for simulating single cell transcriptional data as well [7,9,10,18,27]. Next, we describe the BoolODE framework that we have developed and highlight its differences with GeneNetWeaver.…”

“…Although a few reviews of GRN inference methods have appeared [24][25][26], there has been only one quantitative evaluation of these algorithms [27]. This paper compared eight approaches of which six were originally developed or use methods created for bulk transcriptome data.…”