Aggregating Residue-Level Protein Language Model Embeddings with Optimal Transport

NaderiAlizadeh, Navid; Singh, Rohit

doi:10.1101/2024.01.29.577794

Cited by 5 publications

(1 citation statement)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, we chose to take the former approach as it explicitly integrates signal across the length of the protein. We note that while this is a commonly used approach, how to best aggregate sequence-length representations into a fixed dimension embedding is an open problem in language modeling [see Naderi Alizadeh and Singh ( 54 ) for one recently proposed approach]. Converting this pooled embedding into a binary (

) or multiclass (

, 17 possible symmetry classes + “Unknown”) prediction requires an additional classification head.…”

Section: Methodsmentioning

confidence: 99%

Democratizing protein language models with parameter-efficient fine-tuning

Sledzieski,

Kshirsagar,

Baek

et al. 2024

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Proteomics has been revolutionized by large protein language models (PLMs), which learn unsupervised representations from large corpora of sequences. These models are typically fine-tuned in a supervised setting to adapt the model to specific downstream tasks. However, the computational and memory footprint of fine-tuning (FT) large PLMs presents a barrier for many research groups with limited computational resources. Natural language processing has seen a similar explosion in the size of models, where these challenges have been addressed by methods for parameter-efficient fine-tuning (PEFT). In this work, we introduce this paradigm to proteomics through leveraging the parameter-efficient method LoRA and training new models for two important tasks: predicting protein–protein interactions (PPIs) and predicting the symmetry of homooligomer quaternary structures. We show that these approaches are competitive with traditional FT while requiring reduced memory and substantially fewer parameters. We additionally show that for the PPI prediction task, training only the classification head also remains competitive with full FT, using five orders of magnitude fewer parameters, and that each of these methods outperform state-of-the-art PPI prediction methods with substantially reduced compute. We further perform a comprehensive evaluation of the hyperparameter space, demonstrate that PEFT of PLMs is robust to variations in these hyperparameters, and elucidate where best practices for PEFT in proteomics differ from those in natural language processing. All our model adaptation and evaluation code is available open-source at https://github.com/microsoft/peft_proteomics . Thus, we provide a blueprint to democratize the power of PLM adaptation to groups with limited computational resources.

show abstract

) or multiclass (

, 17 possible symmetry classes + “Unknown”) prediction requires an additional classification head.…”

Section: Methodsmentioning

confidence: 99%

Democratizing protein language models with parameter-efficient fine-tuning

Sledzieski,

Kshirsagar,

Baek

et al. 2024

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

ProteinCLIP: enhancing protein language models with natural language

Wu,

Chang,

Zou

2024

Preprint

View full text Add to dashboard Cite

Language models have enabled a new era of biological sequence modeling. However, extracting meaningful sequence-level embeddings from these models remains challenging. In this work, we introduce ProteinCLIP, which applies contrastive learning between a protein's amino acid sequence and curated text describing its function. ProteinCLIP thus learns to take a pre-trained protein language model's sequence embedding and refines it produce a function-centric embedding. We show that this embedding space yields sequence representations that enable state-of-the-art performance across a variety of important yet challenging tasks in the study of proteins -- from predicting protein protein interactions to accurately detecting homologous proteins despite low sequence similarity. More broadly, ProteinCLIP demonstrates the effectiveness of multi-modal learning in biological contexts, and how such strategies can help isolate key signals from large models and further improve their utility.

show abstract

Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks

Ko,

Parkinson,

Wang

2024

Preprint

View full text Add to dashboard Cite

Protein language models (pLMs) have traditionally been trained in an unsupervised manner using large protein sequence databases with an autoregressive or masked-language modeling training paradigm. Recent methods have attempted to enhance pLMs by integrating additional information, in the form of text, which are referred to as “text+protein” language models (tpLMs). We evaluate and compare six tpLMs (OntoProtein, ProteinDT, ProtST, ProteinCLIP, ProTrek, and ESM3) against ESM2, a baseline text-free pLM, across six downstream tasks designed to assess the learned protein representations. We find that while tpLMs outperform ESM2 in five out of six benchmarks, no tpLM was consistently the best. Thus, we additionally investigate the potential of embedding fusion, exploring whether the combinations of tpLM embeddings can improve performance on the benchmarks by exploiting the strengths of multiple tpLMs. We find that combinations of tpLM embeddings outperform single tpLM embeddings in five out of six benchmarks, highlighting its potential as a useful strategy in the field of machine-learning for proteins. To facilitate the practical application of embedding fusion, we outline a heuristic framework to efficiently identify the optimal combination of embeddings, reducing the exponential time complexity of an exhaustive combination search down to a manageable linear time complexity. Using our embedding fusion framework, we achieve state-of-the-art performances on the protein-protein interaction prediction and homologous sequence recovery tasks without any specific model adjustments or hyperparameter tuning. Our experiments suggest embedding fusion is a useful tool in the machine-learning for proteins toolbox. Lastly, this study highlights the potential of future research on additional strategies for maximizing the utility of pLMs.

show abstract

Aggregating Residue-Level Protein Language Model Embeddings with Optimal Transport

Cited by 5 publications

References 50 publications

Democratizing protein language models with parameter-efficient fine-tuning

Democratizing protein language models with parameter-efficient fine-tuning

ProteinCLIP: enhancing protein language models with natural language

Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks

Contact Info

Product

Resources

About