A deep unsupervised language model for protein design

Abstract

Protein design aims to build new proteins from scratch, thereby holding the potential to tackle many environmental and biomedical problems. Recent progress in the field of natural language processing (NLP) has enabled the implementation of ever-growing language models capable of understanding and generating text with human-like capabilities. Given the many similarities between human languages and protein sequences, the use of NLP models offers itself for predictive tasks in protein research. Motivated by the evident success of generative Transformer-based language models such as the GPT-x series, we developed ProtGPT2, a language model trained on protein space that generates de novo protein sequences that follow the principles of natural ones. In particular, the generated proteins display amino acid propensities which resemble natural proteins. Disorder and secondary structure prediction indicate that 88% of ProtGPT2-generated proteins are globular, in line with natural sequences. Sensitive sequence searches in protein databases show that ProtGPT2 sequences are distantly related to natural ones, and similarity networks further demonstrate that ProtGPT2 is sampling unexplored regions of protein space. AlphaFold prediction of ProtGPT2-sequences yielded well-folded non-idealized structures with embodiments as well as large loops and revealed new topologies not captured in current structure databases. ProtGPT2 has learned to speak the protein language. It has the potential to generate de novo proteins in a high throughput fashion in a matter of seconds. The model is easy-to-use and freely available.

Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem

“…We next cover additional related work on generative models of proteins sequence and structure, beyond the discussion in Section 1.1. Following the success of deep language models, Ferruz et al [11] developed protein sequence models to generate new proteins, but these models do not allow specification of structural motifs. Another class of methods, referred to as fixed backbone sequence design [12,20,40,27,18], attempts to solve the problem of identifying a sequence that folds into any given designable backbone structure.…”

Convolutions are competitive with transformers for protein sequence pretraining

“…Large pretrained protein language models, largely relying on the attention-based transformer [Vaswani et al, 2017] architecture, have advanced the ability of machine-learning methods to predict protein structure and function from sequence, especially when labeled training data is sparse. Most modern self-supervised protein sequence pretraining combines a transformer model with either an autoregressive likelihood [Madani et al, 2020, 2021, Ferruz et al, 2022, Hesslow et al, 2022] or with the masked language modeling (MLM) task introduced for natural language by BERT (bidirectional encoder representations from transformers) [Devlin et al, 2018]. Pretrained transformer protein MLMs contain structural information [Rao et al, 2019, Rives et al, 2021, Chowdhury et al, 2021], encode evolutionary trajectories [Hie et al, 2022a, 2021], are zero-shot predictors of mutation fitness effects [Meier et al, 2021], improve out-of-domain generalization on protein engineering datasets [Dallago et al, 2021], and suggest improved sequences for engineering [Hie et al, 2022b].…”

Masked Inverse Folding with Sequence Transfer for Protein Representation Learning

“…Sequence-only protein language models A large body of work has recently studied the application of language models to sequence-only protein generation [Madani et al, 2020, Shin et al, 2021, Ferruz et al, 2022, Hesslow et al, 2022 and representation learning. See Wu et al [2021] for a more comprehensive review of de novo protein sequence design with deep generative models.…”