DNA language models are powerful predictors of genome-wide variant effects

Benegas, Gonzalo; Batra, Sanjit Singh; Song, Yun S.

doi:10.1101/2022.08.22.504706

Cited by 33 publications

(62 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work has established MLM, specifically masked nucleotide modeling, as a method to pre-train models for genomics (Ji et al ., 2021; Benegas et al ., 2022). It was shown that these models capture some semantics of the regulatory code.…”

Section: Introductionmentioning

confidence: 99%

Species-aware DNA language models capture regulatory elements and their evolution

Gankin

Karollus

Grosshauser

et al. 2023

Preprint

View full text Add to dashboard Cite

Motivation: Predicting gene expression from DNA is an open field of research. As in many areas, labeled data is dwarfed by unlabelled data, i.e. species with a sequenced genome but no gene expression assay data. Pretraining on unlabelled data using masked language modeling has proven highly successful in overcoming data constraints in natural language and proteomics. However, in genomics, this approach has so far been applied only to single genomes, neither leveraging conservation of regulatory sequences across species nor the vast amount of available genomes. Results: Here we train a masked language model on more than 800 species spanning over 500 million years of evolution. We show that explicitly modeling species is instrumental in capturing conserved yet evolving regulatory elements and in controlling for oligomer biases. We extract embeddings for 3' untranslated regions of Saccharomyces cerevisiae and Schizosaccharomyces pombe and use them to achieve prediction of mRNA half-life that is better or on-par with the state-of-the-art, demonstrating the utility of the approach for regulatory genomics. Moreover, we show that the per-base reconstruction probability of our model significantly predicts RNA-binding protein bound sites directly. Altogether, our work establishes a self-supervised framework to leverage large genome collections of evolutionary distant species for regulatory genomics and contributes to alignment-free comparative genomics. Availability and implementation: The source code and trained models are available at: https://github.com/DennisGankin/species-aware-DNA-LM .

show abstract

Section: Introductionmentioning

confidence: 99%

Species-aware DNA language models capture regulatory elements and their evolution

Gankin

Karollus

Grosshauser

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides, we may employ distill learning techniques to transfer the knowledge in LLMs to lightweight architectures like CNN to handle long sequences. Besides, given recent studies have shown that some improved versions of CNNs can achieve superior performance to Transformer-based models (49, 50), it is also promising to develop pre-trained genomic sequence language models without Transformer (51).…”

Section: Discussionmentioning

confidence: 99%

Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

Chen

Zhou

Ding

et al. 2023

Preprint

View full text Add to dashboard Cite

RNA splicing is an important post-transcriptional process of gene expression in eukaryotic organisms. Here, we developed a novel language model, SpliceBERT, pre-trained on the precursor messenger RNA sequences of 72 vertebrates to improve sequence-based modelling of RNA splicing. SpliceBERT is capable of generating embeddings that preserve the evolutionary information of nucleotides and functional characteristics of splice sites. Moreover, the pre-trained model can be utilized to prioritize potential splice-disrupting variants in an unsupervised manner based on genetic variants' impact on the output of SpliceBERT for sequence context. Benchmarked on a multi-species splice site and a human branchpoint prediction task, SpliceBERT outperformed not only conventional baseline models but also other language models pretrained only on the human genome. Our study highlighted the importance of unsupervised learning with genomic sequences of multiple species and indicated that language models were promising approaches to decipher the determinants of RNA splicing.

show abstract

“…As we have shown here, researchers do not need to start from scratch and can repurpose existing networks for evolutionary applications. Unsurprisingly, such repurposing has led to several creative purposes well outside of the developer's initial intention, such as applying cycleGANs [72] to normalize single cell RNA-seq data across datasets [73] and using natural language processing models to predict the effect of nucleotide variants [74]. At a time when unprecedented resources are being used to develop deep-learning technologies outside of biology, biologists from all fields should consider the myriad ways in which these technologies can be leveraged to solve problems and answer longstanding questions.…”

Section: Discussionmentioning

confidence: 99%

“…2017) to normalize single cell RNA-seq data across datasets (Khan et al . 2022) and using natural language processing models to predict the effect of nucleotide variants (Benegas et al . 2022).…”

Section: Discussionmentioning

confidence: 99%

This population does not exist: learning the distribution of evolutionary histories with generative adversarial networks

Booker

Ray

Schrider

2022

Preprint

View full text Add to dashboard Cite

Numerous studies over the last decade have demonstrated the utility of machine learning methods when applied to population genetic tasks. More recent studies show the potential of deep learning methods in particular, which allow researchers to approach problems without making prior assumptions about how the data should be summarized or manipulated, instead learning their own internal representation of the data in an attempt to maximize inferential accuracy. One type of deep neural network, called Generative Adversarial Networks (GANs), can even be used to generate new data, and this approach has been used to create individual artificial human genomes free from privacy concerns. In this study, we further explore the application of GANs in population genetics by designing and training a network to learn the statistical distribution of population genetic alignments (i.e. data sets consisting of sequences from an entire population sample) under several diverse evolutionary histories—the first GAN capable of performing this task. After testing multiple different neural network architectures, we report the results of a fully differentiable Deep-Convolutional Wasserstein GAN with gradient penalty that is capable of generating artificial examples of population genetic alignments that successfully mimic key aspects of the training data, including the site frequency spectrum, differentiation between populations, and patterns of linkage disequilibrium. We demonstrate consistent training success across various evolutionary models, including models of panmictic and subdivided populations, populations at equilibrium and experiencing changes in size, and populations experiencing either no selection or positive selection of various strengths, all without the need for extensive hyperparameter tuning. Overall, our findings highlight the ability of GANs to learn and mimic population genetic data, and suggest future areas where this work can be applied in population genetics research that we discuss herein.AUTHOR SUMMARYThe application of deep-learning to biological problems has expanded greatly over the last decade. One type of deep neural network, called a Generative Adversarial Network (GAN), attempts to generate artificial examples of a given type of data by learning to fool a discriminator that is simultaneously learning to discriminate between real and artificial examples. In this study, we design a GAN whose purpose is to generate artificial examples of genetic alignments from biological populations of varying evolutionary histories—essentially learning the statistical distribution of those evolutionary histories. We show that our GAN is able to mimic key aspects of the genetic alignments relevant to population genetics, and that the GAN does not require extensive tuning of the network parameters. Ultimately, this work demonstrates the ability of these networks to learn and mimic population genetic data, and highlights future areas where this work can be applied and expanded.

show abstract

DNA language models are powerful predictors of genome-wide variant effects

Cited by 33 publications

References 77 publications

Species-aware DNA language models capture regulatory elements and their evolution

Species-aware DNA language models capture regulatory elements and their evolution

Self-supervised learning on millions of pre-mRNA sequences improves sequence-based RNA splicing prediction

This population does not exist: learning the distribution of evolutionary histories with generative adversarial networks

Contact Info

Product

Resources

About