Model-based feature selection and clustering of RNA-seq data for unsupervised subtype discovery

Abstract

Clustering is a form of unsupervised learning that aims to uncover latent groups within data based on similarity across a set of features. A common application of this in biomedical research is in delineating novel cancer subtypes from patient gene expression data, given a set of informative genes. However, it is typically unknown a priori what genes may be informative in discriminating between clusters, and what the optimal number of clusters are. Few methods exist for performing unsupervised clustering of RNA-seq samples, and none currently adjust for between-sample global normalization factors, select cluster-discriminatory genes, or account for potential confounding variables during clustering. To address these issues, we propose the Feature Selection and Clustering of RNA-seq (FSCseq): a model-based clustering algorithm that utilizes a finite mixture of regression (FMR) model and utilized the quadratic penalty method with a SCAD penalty. The maximization is done by a penalized Classification EM algorithm, allowing us to include normalization factors and confounders in our modeling framework. Given the fitted model, our framework allows for subtype prediction in new patients via posterior probabilities of cluster membership. Based on simulations and real data analysis, we show the advantages of our method relative to competing approaches.

ZINBMM: a general mixture model for simultaneous clustering and gene selection using single-cell transcriptomic data

“…As in perhaps all studies, the choice of the candidate tuning parameters cannot be fully objective. The adopted strategy has been very common in published studies [ 6 , 21 ]. Such a choice of candidate values has led to satisfactory performance in our numerical studies.…”

“…Moreover, the selected cell type-specific genes may additionally assist in understanding biological cell types. Examples of such developments are FSCseq [ 21 ] and snbClust [ 22 ], which are two negative binomial (NB) mixture model-based methods and perform feature selection using the penalisation technique. However, these models do not consider dropouts and may be ineffective with highly sparse data.…”

Integrated Multi-Omic Analysis Reveals Immunosuppressive Phenotype Associated with Poor Outcomes in High-Grade Serous Ovarian Cancer

“…Using all available features as input can negatively impact cluster accuracy and distinction [30]. Often, features are selected based on variabilityusing a measure of mean absolute deviation (MAD) across all samples [31][32][33]. Thus, the most variable features are used as inputs for unsupervised learning.…”

“…Multi-omics clustering methods used for subtyping have included basic k-means clustering, iCluster, similarity network fusion, and consensus clustering algorithms [33,34]. Such methods work well with high-dimensional data but are equally dependent upon the input data used for clustering.…”