Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval

Abstract

The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease likelihood, to predicting immune-escape mutations in viruses and designing novel biotherapeutic proteins. Deep generative models of protein sequences trained on multiple sequence alignments have been the most successful approaches so far to address these tasks. The performance of these methods is however contingent on the availability of sufficiently deep and diverse alignments for reliable training. Their potential scope is thus limited by the fact many protein families are hard, if not impossible, to align. Large language models trained on massive quantities of non-aligned protein sequences from diverse families address these problems and show potential to eventually bridge the performance gap. We introduce Tranception, a novel transformer architecture leveraging autoregressive predictions and retrieval of homologous sequences at inference to achieve state-of-the-art fitness prediction performance. Given its markedly higher performance on multiple mutants, robustness to shallow alignments and ability to score indels, our approach offers significant gain of scope over existing approaches. To enable more rigorous model testing across a broader range of protein families, we develop ProteinGym -an extensive set of multiplexed assays of variant effects, substantially increasing both the number and diversity of assays compared to existing benchmarks.

NeuroFold: A Multimodal Approach to Generating Novel Protein Variantsin silico

“…The next best-performing model was ESM-MSA with 64.18%, with ESM-1v and single sequence AlphaFold2 scoring the lowest in this metric (48.06% and 44.37% respectively). Overall, the accuracy achieved by NeuroFold represents a significant improvement over the typical success rates of ∼14.17% achieved using traditional methods (Notin et al, 2022; Table S2). The precision of NeuroFold on each dataset was also independently computed (Figure 4).…”

“…NeuroFold was tested on experimentally validated proteins from the ProteinGym dataset (Notin et al, 2022), TAPE stability dataset (Rao, 2019; Rocklin et al, 2017), as well as additional datasets collected from various studies (Johnson et al, 2023; Madani et al, 2023; Repecka et al, 2021; Russ et al, 2020). The subset of proteins used from the ProteinGym dataset was carefully curated so as to exclude experimental data from proteins with irrelevant non-enzymatic functions in processes such as viral replication and protein-protein interactions.…”

“…The final dataset consisted of 116,321 individual protein sequences from 38 protein families/taxa (Johnson et al, 2023; Madani et al, 2023; Notin et al, 2022; Rao, 2019; Repecka et al, 2021; Rocklin et al, 2017; Russ et al, 2020), including variants with single point mutations, multiple point mutations, and indels. The benchmarks for fitness within these datasets consist of enzyme activity, enzyme stability, and fluorescence.…”

“…Conventional experimental methods, like deep mutational scans (DMS), prove undesirable for generating improved enzyme variants as they necessitate costly and highly parallelizable assays to be effective while only sampling a small portion of the protein sequence space, typically limited to a few point mutations or indels (Notin et al, 2022). For example, the fitness landscape of point mutations in three indole-3-glycerol phosphate synthase (IGPS) orthologs was assayed in one study to reveal the importance of both sequence and structural features of these model TIM barrels (Chan et al, 2017).…”

VariPred: Enhancing Pathogenicity Prediction of Missense Variants Using Protein Language Models

“…Recent studies have used experimental data to evaluate the performance of PLMs in predicting the functional effects of variants [17,20,21,24,25]. However, to date, only one study ('ESM variant') has used a PLM to predict the clinical significance of a mutation 22 .…”