Subject level clustering using a negative binomial model for small transcriptomic studies

Abstract

BackgroundUnsupervised clustering represents one of the most widely applied methods in analysis of high-throughput ‘omics data. A variety of unsupervised model-based or parametric clustering methods and non-parametric clustering methods have been proposed for RNA-seq count data, most of which perform well for large samples, e.g. N ≥ 500. A common issue when analyzing limited samples of RNA-seq count data is that the data follows an over-dispersed distribution, and thus a Negative Binomial likelihood model is often used. Thus, we have developed a Negative Binomial model-based (NBMB) clustering approach for application to RNA-seq studies.ResultsWe have developed a Negative Binomial Model-Based (NBMB) method to cluster samples using a stochastic version of the expectation-maximization algorithm. A simulation study involving various scenarios was completed to compare the performance of NBMB to Gaussian model-based or Gaussian mixture modeling (GMM). NBMB was also applied for the clustering of two RNA-seq studies; type 2 diabetes study (N = 96) and TCGA study of ovarian cancer (N = 295). Simulation results showed that NBMB outperforms GMM applied with different transformations in majority of scenarios with limited sample size. Additionally, we found that NBMB outperformed GMM for small clusters distance regardless of sample size. Increasing total number of genes with fixed proportion of differentially expressed genes does not change the outperformance of NBMB, but improves the overall performance of GMM. Analysis of type 2 diabetes and ovarian cancer tumor data with NBMB found good agreement with the reported disease subtypes and the gene expression patterns. This method is available in an R package on CRAN named NB.MClust.ConclusionUse of Negative Binomial model based clustering is advisable when clustering over dispersed RNA-seq count data.Electronic supplementary materialThe online version of this article (10.1186/s12859-018-2556-9) contains supplementary material, which is available to authorized users.

Model-Based Feature Selection and Clustering of Rna-Seq Data for Unsupervised Subtype Discovery

“…Following the guidelines by Celeux and Govaert (1992) and Rose (1998), we set the annealing rate r = 0.9, similar to other methods (Si et al, 2013;Li et al, 2018). It is suggested that the entropy should be sufficiently large in early iterations, eventually shrinking to 0 in later iterations (Klein and Dubes, 1989;van Laarhoven and Aarts, 1987;Rose, 1998).…”

“…Many studies have demonstrated the clinical utility of such molecular subtypes, where significant differences in patient prognosis and treatment response between subtypes have been observed (Perou et al, 2000;Chia et al, 2012;Mao et al, 2017). However, few methods are able to cluster samples whose genes (features) are measured via RNA-seq, a common platform for measuring gene expression based upon high-throughput sequencing (Mo et al, 2013;Li et al, 2018). Keywords and phrases: clustering, RNA-seq, confounders, batch Gene expression in RNA-seq studies is typically quantified in terms of genelevel read counts, defined by the number of sequencing reads mapping to protein coding regions in a particular gene following sequence alignment.…”

“…We utilize a gene-gene independence assumption, which has been successfully implemented in prior RNA-seq read count-based clustering methods (Si et al, 2013;Mo et al, 2013;Li et al, 2018), as a reasonable approximation to simplify (1). Under this assumption,…”

Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery

“…However, few methods are able to cluster samples whose genes (features) are measured via RNA-seq, a common platform for measuring gene expression based upon high-throughput sequencing (Mo et al, 2013;Li et al, 2018).…”

“…iCluster+ also performs feature selection by inducing sparsity via the lasso penalty, but ignores potential overdispersion, or extra-Poisson variation, in counts. NBMB (Li et al, 2018) accounts for overdispersion via the negative-binomial distribution, but cannot identify clusterdiscriminatory genes. In addition, neither method is able to adjust for confounding factors, such as batch effects or differences in sequencing depth.…”