Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model

Abstract

Understanding protein sequences is vital and urgent for biology, healthcare, and medicine. Labeling approaches are expensive yet time-consuming, while the amount of unlabeled data is increasing quite faster than that of the labeled data due to low-cost, high-throughput sequencing methods. In order to extract knowledge from these unlabeled data, representation learning is of significant value for protein-related tasks and has great potential for helping us learn more about protein functions and structures. The key problem in the protein sequence representation learning is to capture the co-evolutionary information reflected by the interresidue co-variation in the sequences. Instead of leveraging multiple sequence alignment as is usually done, we propose a novel method to capture this information directly by pre-training via a dedicated language model, i.e., Pairwise Masked Language Model (PMLM). In a conventional masked language model, the masked tokens (i.e. amino acid residues) are modeled by conditioning on the unmasked tokens only, but processed independently to each other. However, our proposed PMLM takes the dependency among masked tokens into consideration, i.e., the probability of a token pair is not equal to the product of the probability of the two tokens. By applying this model, the pre-trained encoder is able to generate a better representation for protein sequences. Our result shows that the proposed method can effectively capture the inter-residue correlations and improves the performance of contact prediction by up to 9% compared to the MLM baseline under the same setting. The proposed model also significantly outperforms the MSA baseline by more than 7% on the TAPE contact prediction benchmark when pretrained on a subset of the sequence database which the MSA is generated from, revealing the potential of the sequence pre-training method to surpass MSA based methods in general.

Convolutions are competitive with transformers for protein sequence pretraining

“…We hope that this work is the first step in investigating the independent and interaction effects of pretraining and architecture for protein sequence modeling. While we evaluate the effects of masked language model pretraining, transformers have also been used for autoregressive language model pretraining (Madani et al, 2020) and pairwise masked language modeling (He et al, 2021), and combining structural information (Mansoor et al, 2021; Zhang et al, 2022; McPartlon et al, 2022; Hsu et al, 2022; Chen et al, 2022; Wang et al, 2022) or functional annotations (Brandes et al, 2021) offers further directions for protein pretraining tasks.…”

Retrieved Sequence Augmentation for Protein Representation Learning

“…Following Rao et al (2019) and Rao et al (2021), we compare our model with vanilla protein representation models, including LSTM(Liu, 2017), Transformers(Vaswani et al, 2017) and pre-trained models ESM-1b(Rives et al, 2019), ProtBERT(Elnaggar et al, 2020). We also compare with state-of-the-art knowledge-augmentation models: Potts Model(Balakrishnan et al, 2011), MSA Transformer(Rao et al, 2021) that inject evolutionary knowledge through MSA, OntoProtein(Zhang et al, 2022) that uses gene ontology knowledge graph to augment protein representations and PMLM(He et al, 2021b) that uses pair-wise pretraining to improve co-evolution awareness. We use the reported results of LSTM from Zhang et al (2021); Xu et al (2022).…”

“…Large scale pre-training enables language models to learn structural and evolutionary knowledge (Elnaggar et al, 2021; Jumper et al, 2021; Lin et al, 2022). Despite these successes, many important applications still require MSAs and other external knowledge (Rao et al, 2021; Jumper et al, 2021; He et al, 2021b; Zhang et al, 2021; Ju et al, 2021; Rao et al, 2020). MSAs have been shown effective in improving representation learning, despite being extremely slow and costly in computation.…”

Structure-informed Language Models Are Protein Designers

“…There is growing interest in developing protein language models ( p LMs) at the scale of evolution due to the abundance of 1D amino acid sequences, such as the series of ESM (Rives et al, 2019; Lin et al, 2022), TAPE (Rao et al, 2019), ProtTrans (Elnaggar et al, 2021), PRoBERTa (Nambiar et al, 2020), PMLM (He et al, 2021), ProteinLM (Xiao et al, 2021), PLUS (Min et al, 2021), Adversarial MLM (McDermott et al, 2021), ProteinBERT (Brandes et al, 2022), CARP (Yang et al, 2022a) in masked language modeling (MLM) fashion, ProtGPT2 (Ferruz et al, 2022) in causal language modeling fashion, and several others (Melnyk et al, 2022a; Madani et al, 2021; Unsal et al, 2022; Nourani et al, 2021; Lu et al, 2020; Sturmfels et al, 2020; Strodthoff et al, 2020). These protein language models are able to generalize across a wide range of downstream applications and can capture evolutionary information about secondary and tertiary structures from sequences alone.…”