Deciphering the language of antibodies using self-supervised learning

Abstract

An individual’s B cell receptor (BCR) repertoire encodes information about past immune responses, and potential for future disease protection. Deciphering the information stored in BCR sequence datasets will transform our fundamental understanding of disease and enable discovery of novel diagnostics and antibody therapeutics. One of the grand challenges of BCR sequence analysis is the prediction of BCR properties from their amino acid sequence alone. Here we present an antibody-specific language model, AntiBERTa, which provides a contextualised representation of BCR sequences. Following pre-training, we show that AntiBERTa embeddings learn biologically relevant information, generalizable to a range of applications. As a case study, we demonstrate how AntiBERTa can be fine-tuned to predict paratope positions from an antibody sequence, outperforming public tools across multiple metrics. To our knowledge, AntiBERTa is the deepest protein family-specific language model, providing a rich representation of BCRs. AntiBERTa embeddings are primed for multiple downstream tasks and can improve our understanding of the language of antibodies.

In silico proof of principle of machine learning-based antibody design at unconstrained scale

“…Therefore, our framework allows for optimizing the generative output of deep learning approaches in future benchmarking studies. 11 , 62 , 63 In addition, further research is needed to understand the relationship between signal (pattern) complexity, encoding and embedding, 20 and the number of sequences needed for achieving satisfactory generation quality.…”

Machine Learning Approaches to TCR Repertoire Analysis

“…Therefore, the models are not evaluated in terms of which downstream tasks can be applied via transfer learning. Recently, attempts appear, which utilize large language models in repertoire analysis (133)(134)(135)(136)(137)180). In AntiBERTa (137), fine-tuning for a downstream task is also investigated.…”

“…Some papers perform pre-training on their own on the repertoire sequencing dataset. In Leem et al (136), each amino acid in a TCR is treated as a word, and a TCR is treated as a sentence to pre-train a BERT language model (AntiBERTa). AntiBERTa achieved a higher ROC-AUC in a paratope prediction task than other tools.…”

Progress and challenges for the machine learning-based design of fit-for-purpose monoclonal antibodies

“…Accumulating evidence reveals that many BERT models can be pruned without impacting their predictive prowess, i.e., most heads in the same layers converge to a similar attention pattern, and thus many layers can be consolidated into a single head. 203 , 204 In biology, attention layers of transformer-based models, including BERT, have been shown to capture long-range interaction in protein and antibody folding by folding AAs that are distant in 1D sequence but spatially adjacent in the 3D structure, to identify active sites and to capture the hierarchy of complex biophysical properties with increasing layer depths 100 , 205 – properties that are also critical for antibody-antigen binding. Nevertheless, as in NLP, these models remain susceptible to overparameterization and lack of interpretability.…”