scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI

Abstract

Generative pre-trained models have achieved remarkable success in various domains such as natural language processing and computer vision. Specifically, the combination of large-scale diverse datasets and pre-trained transformers has emerged as a promising approach for developing foundation models. While texts are made up of words, cells can be characterized by genes. This analogy inspires us to explore the potential of foundation models for cell and gene biology. By leveraging the exponentially growing single-cell sequencing data, we present the first attempt to construct a single-cell foundation model through generative pre-training on over 10 million cells. We demonstrate that the generative pre-trained transformer, scGPT, effectively captures meaningful biological insights into genes and cells. Furthermore, the model can be readily fine-tuned to achieve state-of-the-art performance across a variety of downstream tasks, including multi-batch integration, multi-omic integration, cell-type annotation, genetic perturbation prediction, and gene network inference. The scGPT codebase is publicly available at https://github.com/bowang-lab/scGPT.

Assessing the limits of zero-shot foundation models in single-cell biology

“…AvgBIO is the arithmetic mean of three individual metrics: ASW, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI), as defined in [6]. NMI and ARI are calculated based on Louvain clusters generated directly from the embedding space [22, 6]. AvgBIO is normalized to a 0-1 scale, with higher values indicating better alignment between clusters and ground truth labels.…”

“…To evaluate the performance of scGPT in its pretraining objective, we used the mean squared error (MSE), as used by the authors for the model’s loss [6]. To evaluate Geneformer’s performance in its pretraining objective, we measured the Pearson’s correlation between the true and predicted ranked lists.…”

“…In this study, we assessed the zero-shot performance of two proposed foundation models in single-cell biology: Geneformer [5] and scGPT [6]. We selected these models as representative examples in a rapidly evolving field that includes other approaches like scBERT [4], scFoundation [7], SCimilarity [8], and GeneCompass [9].…”

Artificial intelligence for neurodegenerative experimental models

“…Similar transfer learning approaches have been used to successfully predict the effect of novel drug-induced perturbations in cancer cell lines from previously learned latent embeddings. 173,174 Recently, several models trained on a large corpus of scRNA-seq data (eg, single-cell Generative Pre-Trained Transformer [scGPT], 175 single-cell bidirectional encoder representations from transformers [scBERT], 176 Geneformer 177 ), have been put forward as generalist base models to be fine-tuned by users with smaller, more targeted datasets. The goal of such resources is to make transfer learning both more robust and easily accessible to the research community.…”

“…To keep up with the pace at which AI is advancing, rather than wait for large uniform datasets to be created, researchers should focus on developing novel, composite methods for large heterogeneous datasets, integrated from different sources, such as those recently developed in the single-cell genomics field. [175][176][177]37,38 Focusing on developing methods that do not rely on high-dimensional uniform data will ensure experimental research into neurodegenerative disease advances alongside AI.…”