xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data

Gong, Jianya; Hao, Minsheng; Zeng, Xin; Liu, Chi-Ming; Ma, Jianzhu; Cheng, Xingyi; Wang, Taifeng; Zhang, Xuegong; Song, Le

doi:10.1101/2023.03.24.534055

Cited by 12 publications

(9 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we do not exhaustively test all available gene transcriptional profiling-based models. Because some models are not open-source [14] [36], directly converting certain foundation models into gene expression signatures is challenging [12] [11]. Taking various factors into consideration, we select scGPT [8], but this does not imply that other large-scale models perform worse.…”

Section: Discussionmentioning

confidence: 99%

GexMolGen: Cross-modal Generation of Hit-like Molecules via Large Language Model Encoding of Gene Expression Signatures

Cheng,

Pan,

Fang

et al. 2023

Preprint

View full text Add to dashboard Cite

Designing hit-like molecules from gene expression signatures takes into account multiple targets and complex biological effects, enabling the discovery of multi-target drugs for complex diseases. Traditional methods relying on similarity searching against a database are limited by the quality and size of the databases. Instead, multimodal deep learning offers the potential to overcome this bottleneck. Additionally, the recent development of foundation models provides a new perspective to understand gene expression levels. Thus, we propose GexMolGen (Gene Expression-based Molecule Generator) based on a foundation model scGPT to generate hit-like molecules from gene expression differences. By taking the desired and control gene expression profile as inputs, GexMolGen designs molecules that can induce the required transcriptome profile. The molecules generated by GexMolGen exhibit a high similarity to known gene inhibitors. Overall, GexMolGen explores the chemical and biological relationships in the drug discovery process. The source code and demo are available athttps://github.com/Bunnybeibei/GexMolGen.

show abstract

Section: Discussionmentioning

confidence: 99%

GexMolGen: Cross-modal Generation of Hit-like Molecules via Large Language Model Encoding of Gene Expression Signatures

Cheng,

Pan,

Fang

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We developed xTrimoGene, a scalable transformer-based model with both algorithmically efficient and engineering acceleration strategies 14 . It included an embedding module and an asymmetric encoder-decoder structure.…”

Section: The Scfoundation Pre-training Frameworkmentioning

confidence: 99%

Large Scale Foundation Model on Single-cell Transcriptomics

Hao

Gong²,

Zeng³

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Large-scale pretrained models have become foundation models, leading to breakthroughs in natural language processing and related fields. Developing foundation models in life science, aimed at deciphering the "languages" of cells and facilitating biomedical research, is challenging yet promising. We developed a large-scale pretrained model, scFoundation, for this purpose. scFoundation was trained on over 50 million human single-cell transcriptomics data, which contain high-throughput observations on the complex molecular features in all known types of cells. scFoundation is currently the largest model in terms of the size of trainable parameters, dimensionality of genes and the number of cells used in the pre-training. Experiments showed that scFoundation can serve as a foundation model for single-cell transcriptomics and can achieve state-of-the-art performances in a diverse array of downstream tasks, such as gene expression enhancement, tissue drug response prediction, single-cell drug response classification, and single-cell perturbation prediction.

show abstract

“…pLMs have steadily gained traction across diverse applications for protein design, including antibody engineering 18 and drug discovery 19,20 . In addition, AI models trained on unlabeled single-cell RNA sequencing (scRNA-seq) data have been published and used for cell annotation purposes [21][22][23][24][25][26][27] . Thus, masking models have proven to substantially outperform previous conventional methods in effectiveness and show great potential in biomedical applications.…”

Section: Introductionmentioning

confidence: 99%

Cytometry Masked Autoencoder: An Accurate and Interpretable Automated Immunophenotyper

Kim,

Ionita,

Lee

et al. 2024

Preprint

View full text Add to dashboard Cite

High-throughput single-cell cytometry data are crucial for understanding immune system's involvement in diseases and responses to treatment. Traditional methods for annotating cytometry data, specifically manual gating and clustering, face challenges in scalability, robustness, and accuracy. In this study, we propose a single-cell masked autoencoder (scMAE), which offers an automated solution for immunophenotyping tasks including cell type annotation. The scMAE model is designed to uphold user-defined cell type definitions, thereby facilitating easier interpretation and cross-study comparisons. The scMAE model operates on a pre-train and fine-tune approach. In the pre-training phase, scMAE employs Masked Single-cell Modelling (MScM) to learn relationships between protein markers in immune cells solely based on protein expression, without relying on prior information such as cell identity and cell type-specific marker proteins. Subsequently, the pre-trained scMAE is fine-tuned on multiple specialized tasks via task-specific supervised learning. The pre-trained scMAE addresses the shortcomings of manual gating and clustering methods by providing accurate and interpretable predictions. Through validation across multiple cohorts, we demonstrate that scMAE effectively identifies co-occurrence patterns of bound labeled antibodies, delivers accurate and interpretable cellular immunophenotyping, and improves the prediction of subject metadata status. Specifically, we evaluated scMAE for cell type annotation and imputation at the cellular-level and SARS-CoV-2 infection prediction, secondary immune response prediction against COVID-19, and prediction the infection stage in the COVID-19 progression at the subject-level. The introduction of scMAE marks a significant step forward in immunology research, particularly in large-scale and high-throughput human immune profiling. It offers new possibilities for predicting and interpretating cellular-level and subject-level phenotypes in both health and disease.

show abstract

xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data

Cited by 12 publications

References 43 publications

GexMolGen: Cross-modal Generation of Hit-like Molecules via Large Language Model Encoding of Gene Expression Signatures

GexMolGen: Cross-modal Generation of Hit-like Molecules via Large Language Model Encoding of Gene Expression Signatures

Large Scale Foundation Model on Single-cell Transcriptomics

Cytometry Masked Autoencoder: An Accurate and Interpretable Automated Immunophenotyper

Contact Info

Product

Resources

About