On Pre-trained Language Models for Antibody

Wang, Danqing; Ye, Fei; Zhou, Hao

doi:10.1101/2023.01.29.525793

Cited by 19 publications

(23 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is expected that using machine learning to decode information in adaptive immune receptor repertoires can transform the prediction of immune responses and accelerate the development of vaccines, therapeutics, and diagnostics [36]. Having demonstrated that both Ablang and ESM2 capture important evolutionary information for both BCRs and TCRs, we shifted our focus to examining the predictive accuracy achieved by PLMs in epitope specificity predictions for TCRs and BCRs, both significant open challenges in immunoinformatics [37,24]. One of the primary limitations of current models for specificity prediction tasks is the scarcity of labeled training data.…”

Section: Comparing Generalist and Domain-specific Embedders For Bcr A...mentioning

confidence: 99%

Do Domain-Specific Protein Language Models Outperform General Models on Immunology-Related Tasks?

Deutschmann,

Pelissier,

Weber

et al. 2023

Preprint

View full text Add to dashboard Cite

Deciphering the antigen recognition capabilities by T~cell and B~cell receptors (antibodies) is essential for advancing our understanding of adaptive immune system responses. In recent years, the development of protein language models (PLMs) has facilitated the development of bioinformatic pipelines where complex amino acid sequences are transformed into vectorized embeddings, which are then applied to a range of downstream analytical tasks. With their success, we have witnessed the emergence of domain-specific PLMs tailored to specific proteins, such as immune receptors. Domain-specific models are often assumed to possess enhanced representation capabilities for targeted applications, however, this assumption has not been thoroughly evaluated. In this manuscript, we assess the efficacy of both generalist and domain-specific transformer-based embeddings in characterizing B and T cell receptors. Specifically, we assess the accuracy of models that leverage these embeddings to predict antigen specificity and elucidate the evolutionary changes that B cells undergo during an immune response. We demonstrate that the prevailing notion of domain-specific models outperforming general models requires a more nuanced examination. We also observe remarkable differences between generalist and domain-specific PLMs, not only in terms of performance but also in the manner they encode information. Finally, we observe that the choice of the size and the embedding layer in PLMs are essential model hyperparameters in different tasks. Overall, our analyzes reveal the promising potential of PLMs in modeling protein function while providing insights into their information handling capabilities. We also discuss the crucial factors that should be taken into account when selecting a PLM tailored to a particular task.

show abstract

Section: Comparing Generalist and Domain-specific Embedders For Bcr A...mentioning

confidence: 99%

Do Domain-Specific Protein Language Models Outperform General Models on Immunology-Related Tasks?

Deutschmann,

Pelissier,

Weber

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…There are a few studies to perform pretraining on protein sequences ( Bepler and Berger 2019 ; Wang et al 2023 ; Zhou et al 2023 ). Bepler and Berger (2019) propose to train an LSTM on protein sequences, which could implicitly incorporate structural information from the global structural similarity between proteins and the contact maps for individual proteins, while STEPS uses novel self-supervised tasks to explicitly model protein structure.…”

Section: Related Workmentioning

confidence: 99%

Structure-aware protein self-supervised learning

et al. 2023

View full text Add to dashboard Cite

Motivation Protein representation learning methods have shown great potential to many downstream tasks in biological applications. A few recent studies have demonstrated that the self-supervised learning is a promising solution to addressing insufficient labels of proteins, which is a major obstacle to effective protein representation learning. However, existing protein representation learning is usually pretrained on protein sequences without considering the important protein structural information. Results In this work, we propose a novel structure-aware protein self-supervised learning method to effectively capture structural information of proteins. In particular, a graph neural network (GNN) model is pretrained to preserve the protein structural information with self-supervised tasks from a pairwise residue distance perspective and a dihedral angle perspective, respectively. Furthermore, we propose to leverage the available protein language model pretrained on protein sequences to enhance the self-supervised learning. Specifically, we identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme. We conduct experiments on three downstream tasks: the binary classification into membrane/non-membrane proteins, the location classification into 10 cellular compartments, and the enzyme-catalyzed reaction classification into 384 EC numbers, and these experiments verify the effectiveness of our proposed method. Availability and Implementation The Alphafold2 database is available in https://alphafold.ebi.ac.uk/. The PDB files are available in https://www.rcsb.org/. The downstream tasks are available in https://github.com/phermosilla/IEConv_proteins/tree/master/Datasets. The code of the proposed method is available in https://github.com/GGchen1997/STEPS_Bioinformatics.

show abstract

“…□ Task-adaptive DR Model Pretraining. To pretrain a DR model 𝑔(•; 𝜙) in an unsupervised fashion, we continuously pretrain the language model on the corpus X, using the contrastive learning widely adopted in recent research [11,13,30,36]. Specifically, for each document 𝑑 𝑖 ∈ X, we first split each document into multiple sentences and randomly sample two sentences 𝑑 𝑖,1 , 𝑑 𝑖,2 as the positive pair.…”

Section: Stage-i: Dense Retrieval With Label Namesmentioning

confidence: 99%

Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Scientific document classification is a critical task for a wide range of applications, but the cost of obtaining massive amounts of humanlabeled data can be prohibitive. To address this challenge, we propose a weakly-supervised approach for scientific document classification using label names only. In scientific domains, label names often include domain-specific concepts that may not appear in the document corpus, making it difficult to match labels and documents precisely. To tackle this issue, we propose WanDeR, which leverages dense retrieval to perform matching in the embedding space to capture the semantics of label names. We further design the label name expansion module to enrich the label name representations. Lastly, a self-training step is used to refine the predictions. The experiments on three datasets show that WanDeR outperforms the best baseline by 11.9% on average. Our code will be published at https://github.com/ritaranx/wander.

show abstract

On Pre-trained Language Models for Antibody

Cited by 19 publications

References 50 publications

Do Domain-Specific Protein Language Models Outperform General Models on Immunology-Related Tasks?

Do Domain-Specific Protein Language Models Outperform General Models on Immunology-Related Tasks?

Structure-aware protein self-supervised learning

Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training

Contact Info

Product

Resources

About