Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Abstract

The scientific community is rapidly generating protein sequence information, but only a fraction of these proteins can be experimentally characterized. While promising deep learning approaches for protein prediction tasks have emerged, they have computational limitations or are designed to solve a specific task. We present a Transformer neural network that pre-trains task-agnostic sequence representations. This model is fine-tuned to solve two different protein prediction tasks: protein family classification and protein interaction prediction. Our method is comparable to existing state-of-the art approaches for protein family classification, while being much more general than other architectures. Further, our method outperforms all other approaches for protein interaction prediction. These results offer a promising framework for fine-tuning the pre-trained sequence representations for other protein prediction tasks.to identify functional characteristics is critical to understanding cellular functions as well as developing potential therapeutic applications [4]. Sequence-based methods to computationally infer protein characteristics have been critical for inferring protein function and other characteristics [5]. Thus, the development of computational methods to infer protein characteristics (which we generally describe as "protein prediction tasks") has become paramount in the field of bioinformatics and computational biology. Here, we develop a Transformer neural network to establish task-agnostic representations of protein sequences, and use the Transformer network to solve two protein prediction tasks. Background: Deep LearningDeep learning, a class of machine learning based on the use of artificial neural networks, has recently transformed the field of computational biology and medicine through its application towards long-standing problems such as image analysis, gene expression modeling, sequence variant calling, and putative drug discovery [6,7,8,9,10]. By leveraging deep learning, field specialists have been able to efficiently design and train models without the extensive feature engineering required by previous methods. In applying deep learning to sequence-based protein characterization tasks, we first consider the field of natural language processing (NLP), which aims to analyze human language through computational techniques [11]. Deep learning has recently proven to be a critical tool for NLP, achieving state-ofthe-art performance on benchmarks for named entity recognition, sentiment analysis, question answering, and text summarization, among others [12,13].Neural networks are functions that map one vector space to another. Thus, in order to use them for NLP tasks, we first need to represent words as real-valued vectors. Often referred to as word embeddings, these vector representations are typically "pre-trained" on an auxiliary task for which we have (or can automatically generate) a large amount of training data. The goal of this pre-training is to learn generically useful representations that enc...

BERTMHC: Improves MHC-peptide class II interaction prediction with transformer and multiple instance learning

“…Self-supervised pretraining has been shown to boost model performance for natural language processing and computer vision tasks (Devlin et al , 2019; Chen et al , 2020). Recent research has also shown the potential benefits of self-supervised pretraining on protein related tasks (Rao et al , 2019; Nambiar et al , 2020), such as contact prediction. However, to date, no work has explored self-supervised pretraining on MHC–peptide related tasks.…”

“…Such models are trained to predict words masked out in a sentence or to predict the next word or sentence following some context. Similar techniques have also been applied to proteins (Rao et al , 2019; Nambiar et al , 2020; Heinzinger et al , 2019). Since these models do not require labels to train, they can be trained on very large corpora of protein sequences across many species.…”

Deep Learning in Proteomics

“…Recent studies show that models based on natural language processing inspired techniques such as Transformer, [217] BERT, [218] and GPT-2 [219] can learn features from a large corpus of protein sequences in a self-supervised fashion, with applications in a variety of downstream tasks. [220,221] Besides a linear sequence of amino acids, proteins can also be modeled as a graph to capture both structure and sequence information. Graph neural networks [222] are powerful deep learning architectures for learning representations of nodes and edges from such data.…”

Artificial Intelligence for COVID-19 Drug Discovery and Vaccine Development

“…Natural language processing models, specifically language modeling techniques, have also made an impact in the domain of COVID-19 vaccine discovery. Pre-trained transformers were used to predict protein interaction (Nambiar et al, 2020) and model molecular reactions in carbohydrate chemistry (Pesciullesi et al, 2020), which can be utilized in the process of vaccine development. Chen et al discussed the use-case of an LSTMbased seq-2-seq model for predicting the secondary structure of certain SARS-COV-2 proteins (Karpov et al, 2019) 3 .…”