The structure-fitness landscape of pairwise relations in generative sequence models

Abstract

If disentangled properly, patterns distilled from evolutionarily related sequences of a given protein family can inform their traits - such as their structure and function. Recent years have seen an increase in the complexity of generative models towards capturing these patterns; from sitewise to pairwise to deep and variational. In this study we evaluate the degree of structure and fitness patterns learned by a suite of progressively complex models. We introduce pairwise saliency, a novel method for evaluating the degree of captured structural information. We also quantify the fitness information learned by these models by using them to predict the fitness of mutant sequences and then correlate these predictions against their measured fitness values. We observe that models that inform structure do not necessarily inform fitness and vice versa, contrasting recent claims in this field. Our work highlights a dearth of consistency across fitness assays as well as divergently provides a general approach for understanding the pairwise decomposable relations learned by a given generative sequence model.

Generative AAV capsid diversification by latent interpolation

“…The performance of the IS model is consistent with previous studies where independent effect terms explain a large fraction of trait variability (Otwinowski et al, 2018). Furthermore, our VAE model can be thought of as a mixture model, with each set of initial coordinates emitting a single sequence profile, within which additive effects explain the majority of variation (Dauparas et al, 2019;Marshall et al).…”

Interpretable Pairwise Distillations for Generative Protein Sequence Models

“…In Ref. [13], for example, the authors test many different hyperparameters for variational autoencoders, which might also have an influence on how well the resulting distributions are approximated by pairwise models.…”

Disentanglement of Entropy and Coevolution using Spectral Regularization

“…These model optimize an approximation of the pseudo-likelihood function called self-supervision or masked-language-modelling (31). We suspect the Jacobian of these models can be computed (8) and regularized with LH to promote sparsity in the hidden representations.…”

“…The parameters of these model have been inferred using a plethora of methods such as GREM-LIN (3), plmDCA (4), bmDCA (5), PSICOV (6) and mfDCA (7). This also includes the most recent low-rank reparametizations such as restricted Boltzmann machines or variational autoencoder(8), and self-attention-based models that share MRF parameters across protein families(9). The parameters from these models are used for protein structure prediction (10–14), protein-protein interaction prediction (15–17), protein design(18–20), mutation effect prediction(21, 22), and protein sequences alignment and homology search (23–26).…”