Contrastive Representation Learning for 3D Protein Structures

Hermosilla, Pedro; Ropinski, Timo

doi:10.48550/arxiv.2205.15675

Cited by 12 publications

(16 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As discussed before, learning on 3D structures cannot benefit from these large amounts of sequential data. Due to this fact, model sizes of those GGNNs are therefore limited or overfitting may occur [39]. On the contrary, it can be seen, comparing the number of protein sequences in the UniProt database [19] to the number of known structures in the PDB, over 1700 times more sequences than structures.…”

Section: Methodsmentioning

confidence: 99%

When Geometric Deep Learning Meets Pretrained Protein Language Models

Radev

2023

Preprint

View full text Add to dashboard Cite

Geometric deep learning has recently achieved great success in non-Euclidean domains, and learning on 3D structures of large biomolecules is emerging as a distinct research area. However, its efficacy is largely constrained due to the limited quantity of structural data. Meanwhile, protein language models trained on substantial 1D sequences have shown burgeoning capabilities with scale in a broad range of applications. Nevertheless, no preceding studies consider combining these different protein modalities to promote the representation power of geometric neural networks. To address this gap, we make the foremost step to integrate the knowledge learned by well-trained protein language models into several state-of-the-art geometric networks. Experiments are evaluated on a variety of protein representation learning benchmarks, including protein-protein interface prediction, model quality assessment, protein-protein rigid-body docking, and binding affinity prediction, leading to an overall improvement of 20% over baselines and the new state-of-the-art performance. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin and can be generalized to complex tasks.

show abstract

Section: Methodsmentioning

confidence: 99%

When Geometric Deep Learning Meets Pretrained Protein Language Models

Radev

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…As discussed before, learning on 3D structures cannot beneőt from these large amounts of sequential data. Due to this fact, model sizes of those GGNNs are therefore limited or overőtting may occur [39]. On the contrary, it can be seen, comparing the number of protein sequences in the UniProt database [19] to the number of known structures in the PDB, over 1700 times more sequences than structures.…”

Section: Methodsmentioning

confidence: 99%

When Geometric Deep Learning Meets Pretrained Protein Language Models

Radev

et al. 2023

Preprint

View full text Add to dashboard Cite

Geometric deep learning has recently achieved great success in non-Euclidean domains, and learning on 3D structures of large biomolecules is emerging as a distinct research area. However, its efficacy is largely constrained due to the limited quantity of structural data. Meanwhile, protein language models trained on substantial 1D sequences have shown burgeoning capabilities with scale in a broad range of applications. Nevertheless, no preceding studies consider combining these different protein modalities to promote the representation power of geometric neural networks. To address this gap, we make the foremost step to integrate the knowledge learned by well-trained protein language models into several state-of-the-art geometric networks. Experiments are evaluated on a variety of protein representation learning benchmarks, including protein-protein interface prediction, model quality assessment, protein-protein rigid-body docking, and binding affinity prediction, leading to an overall improvement of 20\% over baselines and the new state-of-the-art performance. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin and can be generalized to complex tasks.

show abstract

“…The pre-trained PLMs have achieved impressive performance on a variety of downstream tasks for structure and function prediction (Rao et al, 2019;Xu et al, 2022c). Recent works have also studied pre-training on unlabeled protein structures for generalizable representations, covering contrastive learning (Zhang et al, 2022;Hermosilla & Ropinski, 2022), self-prediction of geometric quantities (Zhang et al, 2022;Chen et al, 2022) and denoising score matching (Guo et al, 2022;Wu et al, 2022a).…”

Section: Related Workmentioning

confidence: 99%

“…Among these methods, various pre-training approaches (Elnaggar et al, 2021;Rives et al, 2021;Zhang et al, 2022) succeed in learning effective protein representations from amount of available protein sequences or from their experimental/predicted structures. Sequence-based pre-training methods (Elnaggar et al, 2021;Rives et al, 2021) can well acquire co-evolutionary information, and structure-based pre-training methods (Zhang et al, 2022;Hermosilla & Ropinski, 2022) are able to capture protein structural characteristics sufficient for tasks like function prediction and fold classification. These two types of information are both useful to indicate underlying protein functions, and they are complementary to each other.…”

Section: Introductionmentioning

confidence: 99%

Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction

Zhang¹,

Xu²,

Lozano³

et al. 2023

Preprint

View full text Add to dashboard Cite

Pre-training methods on proteins are recently gaining interest, leveraging either protein sequences or structures, while modeling their joint energy landscape is largely unexplored. In this work, inspired by the success of denoising diffusion models, we propose the DiffPreT approach to pretrain a protein encoder by sequence-structure multimodal diffusion modeling. DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the multimodal diffusion trajectory, which acquires the joint distribution of sequences and structures. Considering the essential protein conformational variations, we enhance DiffPreT by a physicsinspired method called Siamese Diffusion Trajectory Prediction (SiamDiff) to capture the correlation between different conformers of a protein. SiamDiff attains this goal by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers. We study the effectiveness of DiffPreT and SiamDiff on both atom-and residue-level structure-based protein understanding tasks. Experimental results show that the performance of DiffPreT is consistently competitive on all tasks, and SiamDiff achieves new state-of-the-art performance, considering the mean ranks on all tasks. The source code will be released upon acceptance.

show abstract

Contrastive Representation Learning for 3D Protein Structures

Cited by 12 publications

References 44 publications

When Geometric Deep Learning Meets Pretrained Protein Language Models

When Geometric Deep Learning Meets Pretrained Protein Language Models

When Geometric Deep Learning Meets Pretrained Protein Language Models

Physics-Inspired Protein Encoder Pre-Training via Siamese Sequence-Structure Diffusion Trajectory Prediction

Contact Info

Product

Resources

About