GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Zvyagin, Max; Brace, Alexander; Hippe, Kyle; Deng, Yuntian; Zhang, Bin; Bohorquez, Cindy Orozco; Clyde, Austin; Kale, Bharat; Perez-Rivera, Danilo; Ma, Huadóng; Mann, Carla M.; Irvin, Michael; Pauloski, J. Gregory; Ward, Logan; Hayot, Valerie; Emani, Murali; Foreman, Sam; Xie, Zhen; Lin, Diangen; Shukla, Maulik; Nie, Weili; Romero, Josh; Dallago, Christian; Vahdat, Arash; Xiao, Chaowei; Gibbs, Thomas; Foster, Ian; Davis, James J.; Papka, Michael E.; Brettin, Thomas; Anandkumar, Anima; Vishwanath, Venkatram; Ramanathan, Arvind

doi:10.1101/2022.10.10.511571

Cited by 26 publications

(21 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have already been early efforts to develop foundation models for biological sequences 33,34 , including RFdiffusion, which generates proteins on the basis of simple specifications (for example, a binding target) 35 . Building on this work, GMAI-based solution can incorporate both language and protein sequence data during training to offer a versatile text interface.…”

Section: Bedside Decision Support Gmai Enables a New Class Of Bedside...mentioning

confidence: 99%

Foundation models for generalist medical artificial intelligence

Moor

Banerjee

Abad

et al. 2023

Nature

544

125

View full text Add to dashboard Cite

The exceptionally rapid development of highly flexible, reusable artificial intelligence (AI) models is likely to usher in newfound capabilities in medicine. We propose a new paradigm for medical AI, which we refer to as generalist medical AI (GMAI). GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities. Here we identify a set of high-impact potential applications for GMAI and lay out specific technical capabilities and training datasets necessary to enable them. We expect that GMAI-enabled applications will challenge current strategies for regulating and validating AI devices for medicine and will shift practices associated with the collection of large medical datasets.

show abstract

Section: Bedside Decision Support Gmai Enables a New Class Of Bedside...mentioning

confidence: 99%

Foundation models for generalist medical artificial intelligence

Moor

Banerjee

Abad

et al. 2023

Nature

544

125

View full text Add to dashboard Cite

show abstract

“…This deluge of data poses both an opportunity and a challenge; on the one hand, abundant genomic data able to uncover the intricate patterns of natural variability across species and populations is vastly available; on the other hand, powerful deep learning methods, that can operate at large scale, are required to perform accurate signal extraction from this large amount of unlabelled genomic data. Large foundation models trained on sequences of nucleotides appear to be the natural choice to seize the opportunity in genomics research, with attempts to explore different angles recently proposed [18, 19, 19, 20].…”

Section: Introductionmentioning

confidence: 99%

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

Dalla-Torre¹,

Gonzalez²,

Revilla³

et al. 2023

Preprint

Self Cite

136

View full text Add to dashboard Cite

Closing the gap between measurable genetic information and observable traits is a longstanding challenge in genomics. Yet, the prediction of molecular phenotypes from DNA sequences alone remains limited and inaccurate, often driven by the scarcity of annotated data and the inability to transfer learnings between prediction tasks. Here, we present an extensive study of foundation models pre-trained on DNA sequences, named the Nucleotide Transformer, integrating information from 3,202 diverse human genomes, as well as 850 genomes from a wide range of species, including model and non-model organisms. These transformer models yield transferable, context-specific representations of nucleotide sequences, which allow for accurate molecular phenotype prediction even in low-data settings. We show that the representations alone match or outperform specialized methods on 11 of 18 prediction tasks, and up to 15 after fine-tuning. Despite no supervision, the transformer models learnt to focus attention on key genomic elements, including those that regulate gene expression, such as enhancers. Lastly, we demonstrate that utilizing model representations alone can improve the prioritization of functional genetic variants. The training and application of foundational models in genomics explored in this study provide a widely applicable stepping stone to bridge the gap of accurate molecular phenotype prediction from DNA sequence alone.

show abstract

“…To alleviate the problem of insufficient data, self-supervised learning (SSL) strategy utilized in large language models (LLM) like BERT ( 14) can be adopted. The SSL strategy has been utilized to learn representations of protein (15,16), non-coding RNA (17) and prokaryote genome (18) sequences. These models were pre-trained on millions of sequences from a large number of species and therefore captured the evolutionary information that is critical for modelling.…”

Section: Introductionmentioning

confidence: 99%

Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

Chen

Zhou

Ding

et al. 2023

Preprint

View full text Add to dashboard Cite

RNA splicing is an important post-transcriptional process of gene expression in eukaryotic organisms. Here, we developed a novel language model, SpliceBERT, pre-trained on the precursor messenger RNA sequences of 72 vertebrates to improve sequence-based modelling of RNA splicing. SpliceBERT is capable of generating embeddings that preserve the evolutionary information of nucleotides and functional characteristics of splice sites. Moreover, the pre-trained model can be utilized to prioritize potential splice-disrupting variants in an unsupervised manner based on genetic variants' impact on the output of SpliceBERT for sequence context. Benchmarked on a multi-species splice site and a human branchpoint prediction task, SpliceBERT outperformed not only conventional baseline models but also other language models pretrained only on the human genome. Our study highlighted the importance of unsupervised learning with genomic sequences of multiple species and indicated that language models were promising approaches to decipher the determinants of RNA splicing.

show abstract

GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Cited by 26 publications

References 65 publications

Foundation models for generalist medical artificial intelligence

Foundation models for generalist medical artificial intelligence

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics

Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

Contact Info

Product

Resources

About