PhyloTransformer: A Self-supervised Discriminative Model for Mutation Prediction Based on a Multi-head Self-attention Mechanism

Abstract

Although coronaviruses have RNA proofreading functions, a large number of variants still exist as quasispecies. Identified coronaviruses might just be the tip of the iceberg, and potentially more fatal variants of concern (VOCs) may emerge over time. These VOCs may exhibit increased pathogenicity, infectivity, transmissibility, angiotensin-converting enzyme 2 (ACE2) binding affinity, and antigenicity, causing an increased threat to public health. In this article, we developed PhyloTransformer, a Transformer-based self-supervised discriminative model, which can model genetic mutations that may lead to viral reproductive advantage. We trained PhyloTransformer on 1,765,297 severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) sequences to infer fitness advantages, by directly modeling the amino acid sequence mutations. PhyloTransformer utilizes advanced techniques from natural language processing, including the Fast Attention Via positive Orthogonal Random features approach (FAVOR+) and the Masked Language Model (MLM), which enable efficient and accurate intra-sequence dependency modeling over the entire RNA sequence. We measured the prediction accuracy of novel mutations and novel combinations using our method and baseline models that only take local segments as input. We found that PhyloTransformer outperformed every baseline method with statistical significance. In order to identify mutations associated with altered glycosylation that might be favored during viral evolution, we predicted the occurrence of mutations in each nucleotide of the receptor binding motif (RBM) and predicted modifications of N-glycosylation sites. We anticipate that the viral mutations predicted by PhyloTransformer may identify potential mutations of threat to guide therapeutics and vaccine design for effective targeting of future SARS-CoV-2 variants.

Running ahead of evolution - AI based simulation for predicting future high-risk SARS-CoV-2 variants

“…DL language models have also been applied for protein prediction tasks, as common protein motifs and domains can be analogized to words, phrases, and sentences in human language [38–41]. A Transformer-based discriminative model was trained with SARS-CoV-2 genetic sequences to predict potential mutations that may lead to enhanced virus transmissibility [42]. Motivated by the success of masked language models such as BERT [43], we design a pretrained protein language model for comprehensive variant prediction, aiming to simulate circulating viral mutation and predict of potentially risky variants.…”

“…Language models have been used to decipher the genetic sequences of virus. For example, a Transformer-based discriminative model was trained with SARS-CoV-2 genetic sequences to predict potential mutations that may lead to enhanced virus transmissibility (Wu et al 2021). Language models have also been applied for protein prediction tasks, as common protein motifs and domains can be analogized to words, phrases, and sentences in human language (Ofer et al 2021; Trifonov 2009; Strait and Dewey 1996; Yu et al 2019).…”