ProGen: Language Modeling for Protein Generation

Madani, Ali; McCann, Bryan; Naik, Nikhil; Keskar, Nitish Shirish; Anand, Namrata; Eguchi, Raphael R.; Huang, P. G.; Socher, Richard

doi:10.1101/2020.03.07.982272

Cited by 179 publications

(184 citation statements)

References 41 publications

Supporting

Mentioning

181

Contrasting

Order By: Relevance

“…1. Overall, BFD was about eight times larger than the largest data sets used previously [19]. Despite the 8-fold increase in data, the number of tokens increased only fivefold ( Fig.…”

Section: Data For Language Models (Lms)mentioning

confidence: 75%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

434

673

View full text Add to dashboard Cite

Motivation: Natural Language Processing (NLP) continues improving substantially through auto-regressive (AR) and auto-encoding (AE) Language Models (LMs). These LMs require expensive computing resources for self-supervised or un-supervised learning from huge unlabelled text corpora. The information learned is transferred through so-called embeddings to downstream prediction tasks. Computational biology and bioinformatics provide vast gold-mines of structured and sequentially ordered text data leading to extraordinarily successful protein sequence LMs that promise new frontiers for generative and predictive tasks at low inference cost. As recent NLP advances link corpus size to model size and accuracy, we addressed two questions: (1) To which extent can High-Performance Computing (HPC) up-scale protein LMs to larger databases and larger models? (2) To which extent can LMs extract features from single proteins to get closer to the performance of methods using evolutionary information? Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models (BERT and Albert) on 80 billion amino acids from 200 million protein sequences (UniRef100) and one language model (Transformer-XL) on 393 billion amino acids from 2.1 billion protein sequences taken from the Big Fat Database (BFD), today's largest set of protein sequences (corresponding to 22- and 112-times, respectively of the entire English Wikipedia). The LMs were trained on the Summit supercomputer, using 936 nodes with 6 GPUs each (in total 5616 GPUs) and one TPU Pod, using V3-512 cores. Results: We validated the feasibility of training big LMs on proteins and the advantage of up-scaling LMs to larger models supported by more data. The latter was assessed by predicting secondary structure in three- and eight-states (Q3=75-83, Q8=63-72), localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89). Dimensionality reduction revealed that the LM-embeddings from unlabelled data (only protein sequences) captured important biophysical properties of the protein alphabet, namely the amino acids, and their well orchestrated interplay in governing the shape of proteins. In the analogy of NLP, this implied having learned some of the grammar of the language of life realized in protein sequences. The successful up-scaling of protein LMs through HPC slightly reduced the gap between models trained on evolutionary information and LMs. Additionally, our results highlighted the importance of bi-directionality when processing proteins as the uni-directional TransformerXL was outperformed by its bi-directional counterparts. Availability ProtTrans: <a href="https://github.com/agemagician/ProtTrans">https://github.com/agemagician/ProtTrans</a>

show abstract

“…1. Overall, BFD was about eight times larger than the largest data sets used previously [19]. Despite the 8-fold increase in data, the number of tokens increased only fivefold ( Fig.…”

Section: Data For Language Models (Lms)mentioning

confidence: 75%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

434

673

View full text Add to dashboard Cite

show abstract

“…Supervised models of protein function are currently limited by the availability and quality of experimental data but will become increasingly accurate and general as researchers continue to experimentally characterize protein sequence space. Other important machine learning advances relevant to protein engineering include generative modeling to sample non-natural protein sequences [23, 29, 30], language models to learn protein representations from diverse natural sequences [31–34], and strategies to incorporate machine learning predictions into directed evolution experiments [35, 36]. Coupled with optimization methods that allow supervised models to efficiently explore new parts of the sequence space [37–40], these approaches could make possible a new generation of data-driven protein engineering.…”

Section: Discussionmentioning

confidence: 99%

Neural networks to learn protein sequence-function relationships from deep mutational scanning data

Gelman

Romero

Gitter

2020

Preprint

View full text Add to dashboard Cite

The mapping from protein sequence to function is highly complex, making it challenging to predict how sequence changes will affect a protein's behavior and properties. We present a supervised deep learning framework to learn the sequence-function mapping from deep mutational scanning data and make predictions for new, uncharacterized sequence variants. We test multiple neural network architectures, including a graph convolutional network that incorporates protein structure, to explore how a network's internal representation affects its ability to learn the sequence-function mapping. Our supervised learning approach displays superior performance over physics-based and unsupervised prediction methods. We find networks that capture nonlinear interactions and share parameters across sequence positions are important for learning the relationship between sequence and function. Further analysis of the trained models reveals the networks' ability to learn biologically meaningful information about protein structure and mechanism. Our software is available from https://github.com/gitter-lab/nn4dms.

show abstract

“…TAPE created a benchmark of five tasks ranging from remote homology to fluorescence prediction to assess protein representation learning models. In [48,63], autoregressive generative models were trained to predict the functional effect of mutations and generate natural-like proteins. All aforementioned studies have been the subject of rapid development, with Transformer architectures seemingly providing the most promising avenue for future research.…”

Section: Protein Language Modelsmentioning

confidence: 99%

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Vig

Madani

Varshney

et al. 2020

Preprint

Self Cite

200

169

View full text Add to dashboard Cite

Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. Through the lens of attention, we analyze the inner workings of the Transformer and explore how the model discerns structural and functional properties of proteins. We show that attention (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We also present a three-dimensional visualization of the interaction between attention and protein structure. Our findings align with known biological processes and provide a tool to aid discovery in protein engineering and synthetic biology. The code for visualization and analysis is available at https://github.com/salesforce/provis.

show abstract

ProGen: Language Modeling for Protein Generation

Cited by 179 publications

References 41 publications

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Neural networks to learn protein sequence-function relationships from deep mutational scanning data

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Contact Info

Product

Resources

About