LM-GVP: an extensible sequence and structure informed deep learning framework for protein property prediction

Abstract

Proteins perform many essential functions in biological systems and can be successfully developed as bio-therapeutics. It is invaluable to be able to predict their properties based on a proposed sequence and structure. In this study, we developed a novel generalizable deep learning framework, LM-GVP, composed of a protein Language Model (LM) and Graph Neural Network (GNN) to leverage information from both 1D amino acid sequences and 3D structures of proteins. Our approach outperformed the state-of-the-art protein LMs on a variety of property prediction tasks including fluorescence, protease stability, and protein functions from Gene Ontology (GO). We also illustrated insights into how a GNN prediction head can inform the fine-tuning of protein LMs to better leverage structural information. We envision that our deep learning framework will be generalizable to many protein property prediction problems to greatly accelerate protein engineering and drug development.

Convolutions are competitive with transformers for protein sequence pretraining

“…We hope that this work is the first step in investigating the independent and interaction effects of pretraining and architecture for protein sequence modeling. While we evaluate the effects of masked language model pretraining, transformers have also been used for autoregressive language model pretraining (Madani et al, 2020) and pairwise masked language modeling (He et al, 2021), and combining structural information (Mansoor et al, 2021; Zhang et al, 2022; McPartlon et al, 2022; Hsu et al, 2022; Chen et al, 2022; Wang et al, 2022) or functional annotations (Brandes et al, 2021) offers further directions for protein pretraining tasks.…”

Contrastive Representation Learning for 3D Protein Structures

“…For the biological process and cellular component data sets both pre-trained methods, fine-tuned and not, outperform models trained from scratch, being the fine-tuned model the one achieving the highest F max . When compared to other methods pre-trained on large sequence data sets of millions of proteins [49,62], our method outperforms those in the molecular function data set and achieves competitive performance on the other two. When compared to the pre-trained method of Zhang et al [69] on 3D structures, our framework outperforms it on two out of three data sets.…”

“…Results. Performance of other methods are obtained from the works of Wang et al [62] and Zhang et al [69]. For completeness, we include additional comparisons in Tbl.…”

Masked Inverse Folding with Sequence Transfer for Protein Representation Learning

“…While we condition on structure and reconstruct sequence, there are other methods for incorporating protein structural information, such as predicting structure similarity between protein sequences (Bepler & Berger, 2019), corrupting and reconstructing the structure in addition to the sequence (Mansoor et al, 2021; Chen et al, 2022), encoding surface features (Townshend et al, 2019), contrastive learning (Zhang et al, 2022; Cao et al, 2021), or a graph encoder without sequence decoding (Somnath et al, 2021; Fuchs et al, 2020). LM-GVP uses the same architecture as MIF-ST consisting of a pretrained language model feeding into a GNN that encodes backbone structure (Wang et al, 2022). However, in LM-GVP the structure-aware module is used as a finetuned prediction head without any pretraining.…”