Generative pretraining from large-scale transcriptomes: Implications for single-cell deciphering and clinical translation

Abstract

Exponential accumulation of single-cell transcriptomes poses great challenge for efficient assimilation. Here, we present an approach entitled tGPT towards integration of 22.3 million single-cell transcriptomes by modeling gene expression rankings as generative pretraining task. tGPT is conceptually simple in that it autoregressively models the ranking of a gene in the context of its preceding neighbors. We demonstrated the high performance of tGPT on a range of fundamental single-cell analysis tasks and novel applications on bulk tissues. The single-cell clusters and cell lineage trajectories derived from tGPT are highly aligned with known cell labels and states. The feature patterns of tumor bulk tissues learned by tGPT are associated with a wide range of genomic alteration events, prognosis and treatment outcome of immunotherapy. tGPT represents a new analytical paradigm for integrating and deciphering massive amount of transcriptome data and it will facilitate the interpretation and clinical translation of single-cell transcriptomes.

scFormer: A Universal Representation Learning Approach for Single-Cell Data Using Transformers

“…Despite these results, there have been few attempts to adopt the transformer architecture into single-cell biology and applications thereof. Shen et al (2022) use the transformer decoder setup to learn the gene name sequence of highly expressed genes, without considering the actual sequenced expression abundance. This leads to loss of major biological signal, as the expression values are informative of cell state and gene-gene relationships.…”

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis

“…tGPT [34], GeneCompass [35], SCimilarity [36], UCE [37] and CellPLM [38]. Details of these models can be found in Appendices D and E. Furthermore, no studies to date have comprehensively evaluated the utility of these models and provided guidance for model training.…”

“…There is a limited number of robust pre-trained models (known as single-cell LLMs) capable of managing multiple tasks in single-cell research. Some single-cell LLMs focus on cell-type annotation or gene function prediction, including scBERT [28], tGPT [29], CellLM [30], and Geneformer [31], while others aim to create a foundation model in this area that can handle multiple tasks, including scGPT [32], scFoundation [33] and SCimilarity [34]. Details of these models can be found in Appendices D and E. Furthermore, no studies to date have comprehensively evaluated the utility of these models and provide guidance for model training.…”

Bioformers: A Scalable Framework for Exploring Biostates Using Transformers

“…These initial explorations have revealed a specific key challenge: the inability to transfer the insights gained by these models across different applications due to their distinct architectures. To overcome this, a wave of research initiatives [11,12,13,14,15] is underway, aiming to develop a foundational model that first deciphers latent information from unlabeled scRNA-seq data and then adapts this knowledge to various tasks. In scBERT [11], introduced by Yang and team in 2022, represents genes as tokens and employs an advanced transformer mechanism [16] to analyze over 16,000 gene tokens per cell.…”