Evolutionary-scale prediction of atomic level protein structure with a language model

Abstract

Large language models have recently been shown to develop emergent capabilities with scale, going beyond simple pattern matching to perform higher level reasoning and generate lifelike images and text. While language models trained on protein sequences have been studied at a smaller scale, little is known about what they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters, the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn information enabling the prediction of the three-dimensional structure of a protein at the resolution of individual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for sequences with low perplexity that are well understood by the language model. ESMFold inference is an order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic proteins in practical timescales.

Towards a General Intermolecular Binding Affinity Calculator

“…With regard to the entire protein universe, however, further challenges still remain for GIBAC, when it comes to K d calculations involving intrinsically disordered proteins, bacause the flexibility involved makes structural modeling and structure-based K d prediction a rather difficult (if possible) problem, to which an experimental approach is perhaps the only feasible solution in practice [54,[96][97][98][99].…”

LambdaPP: Fast and accessible protein-specific phenotype predictions

“…Further parameters are an early stop criterion of a prediction certainty (pLDDT) above 85 or below 40, a default recycle count of 3 and a compilation of only the best performing out of five AlphaFold2 models. As the goal of LambdaPP is to provide a single reference for pLM-based predictions, 3D structure will soon be predicted using tools just recently presented in the literature (35; 72; 74), which will allow structure prediction to happen in seconds rather than in minutes, at accuracy comparable with MSA-based methods.…”

“…Embeddings from pLMs have been successfully used as input to downstream protein prediction tools (78; 37-39; 41; 42; 44; 64; 14; 25; 27; 63; 72). While some pLM-based methods do not reach the performance of MSA-based methods (23; 38; 72), others exceed those (5; 10; 23; 39; 42; 64; 28; 29; 36). Prediction performance has risen so much that sequence-specific predictions based on pLMs can capture some aspects of structural and functional dynamics better than much more accurate family-averaged solutions even from AlphaFold2 (35; 72; 74).…”

“…All results are linked to 3D structure visualizations, currently retrieved from the AlphaFold Database (71) or if unavailable predicted using ColabFold , easing the access to AlphaFold2 at similar performance (45). As the only method of LambdaPP not based on pLM inputs, it will soon be complemented by pLM-based solutions (35; 72; 74). As novel AI tools leveraging embeddings emerge, e.g.…”

From sequence to function through structure: deep learning for protein design

“…It is thus becoming clear that pLM embeddings are rich inputs to downstream prediction methods of protein function and structure competing with those that exploit MSAs. However, what is encoded in embeddings, including how much evolutionary information as defined in MSAs (i.e., residue co-evolution) is implicitly captured, remains a subject of debate, even in light of correlation between MSAs and pLM embeddings in accuracy for 3D structure prediction [40].…”