Few Shot Protein Generation

Abstract

We present the MSA-to-protein transformer, a generative model of protein sequences conditioned on protein families represented by multiple sequence alignments (MSAs). Unlike existing approaches to learning generative models of protein families, the MSA-to-protein transformer conditions sequence generation directly on a learned encoding of the multiple sequence alignment, circumventing the need for fitting dedicated family models. By training on a large set of well curated multiple sequence alignments in Pfam, our MSA-to-protein transformer generalizes well to protein families not observed during training and outperforms conventional family modeling approaches, especially when MSAs are small. Our generative approach accurately models epistasis and indels and allows for exact inference and efficient sampling unlike other approaches. We demonstrate the protein sequence modeling capabilities of our MSA-to-protein transformer and compare it with alternative sequence modeling approaches in comprehensive benchmark experiments.

Likelihood-based fine-tuning of protein language models for few-shot fitness prediction and design

“…However, unconditional PLMs suffer from a lack of context, often requiring fine-tuning to specialise their distributions towards a particular protein family of interest [25]. As a result, the leading autoregressive models exploit the information in MSAs to improve predictions, either by biasing language model likelihoods with statistics from the MSA, in the case of Tranception [28], or by explicitly conditioning on the MSA [12,33,42].…”

“…Family-based protein language models represent the conditional distribution over family members given a subset of other family members [35,12,33,42]. These models have proved especially effective as zero-shot fitness predictors, due to their ability to explicitly condition on evolutionary context to predict the effects of mutations.…”