Antibody Representation Learning for Drug Discovery

Abstract

Therapeutic antibody development has become an increasingly popular approach for drug development. To date, antibody therapeutics are largely developed using large scale experimental screens of antibody libraries containing hundreds of millions of antibody sequences. The high cost and difficulty of developing therapeutic antibodies create a pressing need for computational methods to predict antibody properties and create bespoke designs. However, the relationship between antibody sequence and activity is a complex physical process and traditional iterative design approaches rely on large scale assays and random mutagenesis. Deep learning methods have emerged as a promising way to learn antibody property predictors, but predicting antibody properties and target-specific activities depends critically on the choice of antibody representations and data linking sequences to properties is often limited. Recently, methods for learning biological sequence representations from large, general purpose protein datasets have demonstrated impressive ability to capture structural and functional properties of proteins. However, existing works have not yet investigated the value, limitations and opportunities of these methods in application to antibody-based drug discovery. In this paper, we present results on a novel SARS-CoV-2 antibody binding dataset and an additional benchmark dataset. We compare three classes of models: conventional statistical sequence models, supervised learning on each dataset independently, and fine-tuning an antibody specific pre-trained embedding model. The pre-trained models were trained on tens of millions of natural healthy human antibody sequences and protein sequences, respectively. Experimental results suggest that self-supervised pretraining of feature representation consistently offers significant im-DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

Machine Learning Optimization of Candidate Antibodies Yields Highly Diverse Sub-nanomolar Affinity Antibody Libraries

“…We separately explored our model performance as a function of the amount of training data and demonstrated additional data, expectedly, results in improved performance [26]. However, after about 7,000 measurements, additional measurements result in less significant performance increases [26]. For this work, we trained our supervised sequence-to-affinity models on all 43,341 measurements that were available to us, but future engineering attempts may optimize use of financial resources by increasing the number of cycles while reducing the number of measurements per cycle.…”

“…We separately explored our model performance as a function of the amount of training data and demonstrated additional data, expectedly, results in improved performance [26]. However, after about 7,000 measurements, additional measurements result in less significant performance increases [26].…”

On Pre-trained Language Models for Antibody

“…Although this paper claims finetuning AntiBERTa for paratope position prediction can achieve state-of-the-art performance, the experimental results lack standard deviations, making it unclear how significant the results obtained are. Recently, Li et al (2022) proposed a antibody-specific language model and explored its performance in SARS-CoV-2 antigen binding, showing context-dependent representations of antibody sequences benefit binding prediction.…”

“…Pretrained Antibody Language Models (PALMs) Encouraged by the success of PLMs in protein representation learning, series work seeks to learn antibody representations based on sequences of antibodies. (Leem et al, 2021;Ruffolo et al, 2021;Olsen et al, 2022b;Prihoda et al, 2022;Li et al, 2022). AntiBERTy (Ruffolo et al, 2021) proposed the first antibody-specific language model, exploring a Transformer trained on 558M natural antibody sequences in the OAS database.…”

De novogeneration of antibody CDRH3 with a pre-trained generative large language model

“…Some other methods utilize antibody sequences, which are relatively cheaper and easier to obtain, to predict affinity for specific antibodies 26,27 . Although such methods employ large-scale pre-trained language models to achieve even more accurate affinity predictions 28,29 , their prediction ability is limited to the trained antigens since they do not consider the antigen information. When dealing with unknown antigens, such as in new mutation affinity tasks like XBB, the lack of antigen information is a significant limitation.…”