RITA: a Study on Scaling Up Generative Protein Sequence Models

Hesslow, Daniel; Zanichelli, Niccoló; Notin, Pascal; Poli, Iacopo; Marks, Debora S.

doi:10.48550/arxiv.2205.05789

Cited by 28 publications

(35 citation statements)

References 22 publications

(32 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter work allows for sequence generation conditioned on tags corresponding to molecular function or taxonomic information. Similar to results in NLP, scaling protein language models to very large sizes seems promising for protein sequences [15].…”

Section: Related Literaturementioning

confidence: 76%

“…We explore different architectural choices, regularization schemes and compare our results with a recently published shallow auto-regressive method [32], which we use as a baseline. We also compare on a smaller scale to fine-tuned large protein language models, using RITA [15], and explore how structural predictions from Alphafold 2 correlate with our results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generating Interacting Protein Sequences using Domain-to-Domain Translation

Meynard-Piganeau

Fabbri

Weigt

et al. 2022

Preprint

View full text Add to dashboard Cite

Being able to artificially design novel proteins of desired function is pivotal in many biological and biomedical applications. Generative statistical modeling has recently emerged as a new paradigm for designing amino acid sequences, including in particular models and embedding methods borrowed from Natural Language Processing (NLP). However, most approaches target single proteins or protein domains, and do not take into account any functional specificity or interaction with the context. To extend beyond current computational strategies, we develop a method for generating protein domain sequences intended to interact with another protein domain. Using data from natural multi-domain proteins, we cast the problem as a translation problem from a given interactor domain to the new domain to be generated, i.e. we generate artificial partner sequences conditional on an input sequence. Evaluating our model's quality using diverse metrics, in part related to distinct biological questions, we show that our method outperforms state-of-the-art shallow auto-regressive strategies.

show abstract

Section: Related Literaturementioning

confidence: 76%

Section: Introductionmentioning

confidence: 99%

Generating Interacting Protein Sequences using Domain-to-Domain Translation

Meynard-Piganeau

Fabbri

Weigt

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These are widely used to perform peptide generative with desired properies. [38][39][40][41][42] Linder et al [43] developed a generative method which can explicitly control sequence diversity during training by penalizing any two generated sequences on the basis of similarity. Ferruz et al [44] developed ProtGPT2, a Transformer-based generative model trained on UniRef50 which can generate de novo protein sequences following the principles of natural ones.…”

Section: A Related Workmentioning

confidence: 99%

“…These are widely used to perform peptide generative with desired properies. 38–42 Linder et al . [43] developed a generative method which can explicitly control sequence diversity during training by penalizing any two generated sequences on the basis of similarity.…”

Section: Introductionmentioning

confidence: 99%

Now What Sequence? Pre-trained Ensembles for Bayesian Optimization of Protein Sequences

Yang

Milas

White

2022

Preprint

View full text Add to dashboard Cite

Pre-trained models have been transformative in natural language, computer vision, and now protein sequences by enabling accuracy with few training examples. We show how to use pre-trained sequence models in Bayesian optimization to design new protein sequences with minimal labels (i.e., few experiments). Pre-trained models give good predictive accuracy at low data and Bayesian optimization guides the choice of which sequences to test. Pre-trained sequence models also obviate the common requirement of finite pools. Any sequence can be considered. We show significantly fewer labeled sequences are required for many sequence design tasks, including creating novel peptide inhibitors with AlphaFold. This work should enable calibrated predictions with few examples and iterative design with low data (1-50).

show abstract

“…Following the principles of DARK3, ProtGPT2 leveraged a GPT2-like model [85] an trained on the Uniref50 dataset [83], leading to a model able to generate proteins in unexplored regions of the natural protein space, while presenting natural-like properties [16]. RITA [86] included a study on the scalability of generative Transformer models with several model-specific (e.g. perplexity) and application-specific (e.g.…”

Section: Introductionmentioning

confidence: 99%

From sequence to function through structure: deep learning for protein design

Ferruz

Heinzinger

Akdel³

et al. 2022

Preprint

View full text Add to dashboard Cite

The process of designing biomolecules, in particular proteins, is witnessing a rapid change in available tooling and approaches, moving from design through physicochemical force fields, to producing plausible, complex sequences fast via end-to-end differentiable statistical models. To achieve conditional and controllable protein design, researchers at the interface of artificial intelligence and biology leverage advances in natural language processing (NLP) and computer vision techniques, coupled with advances in computing hardware to learn patterns from growing biological databases, curated annotations thereof, or both. Once learned, these patterns can be leveraged to provide novel insights into mechanistic biology and the design of biomolecules. However, navigating and understanding the practical applications for the many recent protein design tools is complex. To facilitate this, we 1) document recent advances in deep learning (DL) assisted protein design from the last three years, 2) present a practical pipeline that allows to go from de novo-generated sequences to their predicted properties and web-powered visualization within minutes, and 3) leverage it to suggest a generated protein sequence which might be used to engineer a biosynthetic gene cluster to produce a molecular glue-like compound. Lastly, we discuss challenges and highlight opportunities for the protein design field.

show abstract

RITA: a Study on Scaling Up Generative Protein Sequence Models

Cited by 28 publications

References 22 publications

Generating Interacting Protein Sequences using Domain-to-Domain Translation

Generating Interacting Protein Sequences using Domain-to-Domain Translation

Now What Sequence? Pre-trained Ensembles for Bayesian Optimization of Protein Sequences

From sequence to function through structure: deep learning for protein design

Contact Info

Product

Resources

About