Transformer-based protein generation with regularized latent space optimization

Abstract

The development of powerful natural language models has improved the ability to learn meaningful representations of protein sequences. In addition, advances in high-throughput mutagenesis, directed evolution and next-generation sequencing have allowed for the accumulation of large amounts of labelled fitness data. Leveraging these two trends, we introduce Regularized Latent Space Optimization (ReLSO), a deep transformer-based autoencoder, which features a highly structured latent space that is trained to jointly generate sequences as well as predict fitness. Through regularized prediction heads, ReLSO introduces a powerful protein sequence encoder and a novel approach for efficient fitness landscape traversal. Using ReLSO, we explicitly model the sequence-function landscape of large labelled datasets and generate new molecules by optimizing within the latent space using gradient-based methods. We evaluate this approach on several publicly available protein datasets, including variant sets of anti-ranibizumab and green fluorescent protein. We observe a greater sequence optimization efficiency (increase in fitness per optimization step) using ReLSO compared with other approaches, where ReLSO more robustly generates high-fitness sequences. Furthermore, the attention-based relationships learned by the jointly trained ReLSO models provide a potential avenue towards sequence-level fitness attribution information. Articles NaTURe MachINe INTeLLIgeNceAn alternative to working in the sequence space is to learn a low-dimensional, semantically rich representation of peptides and proteins. These latent representations collectively form the latent space, which is easier to navigate. With this approach, a therapeutic candidate can be optimized using its latent representation, in a procedure called latent space optimization.Here we propose ReLSO, a deep transformer-based approach to protein design, which combines the powerful encoding ability of a transformer model with a bottleneck that produces information-rich, low-dimensional latent representations. The latent space in ReLSO, besides being low dimensional, is regularized to be (1) smooth with respect to structure and fitness by way of fitness prediction from the latent space, (2) continuous and interpolatable between training data points and (3) pseudoconvex on the basis of negative sampling outside the data. This highly designed latent space enables optimization directly in latent space using gradient ascent on the fitness and converges to an optimum that can then be decoded back into the sequence space.Key contributions of ReLSO include the following.

ProT-VAE: Protein Transformer Variational AutoEncoder for Functional Protein Design

“…Henderson and Fehr employed VAEs as an information bottleneck regularizer for transformer embeddings and used this model to embed and generate text within a nonparametric space of mixture distributions [55]. Our work is most closely related to the recent work of Castro et al, who introduced the Regularized Latent Space Optimization (ReLSO) approach for data-driven protein engineering [40]. The jointly trained autoencoder (JT-AE) architecture underpinning this approach comprises a transformer encoder, low-dimensional projection into a latent space bottleneck, 1D convolutional neural network (CNN) decoder, and fully-connected network to predict function from the latent space embedding.…”

From sequence to function through structure: Deep learning for protein design

“…Another model with generative ability is EVE [100] , a VAE used to predict the pathogenicity of protein variants. A particularly interesting application of language models came with ReLSO [101] , which used a transformer autoencoder paired with function prediction, inferring protein functionality from sequence embeddings. This model can also be used to generate new sequences by optimizing the latent space with gradient ascent.…”

Possibilities of Using De Novo Design for Generating Diverse Functional Food Enzymes

“…Protein sequential arrangement or functional motif decomposition was thought to be like human language, which can be organized to represent certain meanings [ 81 ]. Accordingly, Natural Language Processing (NLP) methods originally used for human language translation were applied to resolving protein folding and de novo design tasks, by extracting features from protein sequences [ 22 , 82 ].…”