Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

Xia, Jun; Zheng, Jiangbin; Tan, Cheng; Wang, Ge; Li, Stan Z.

doi:10.1101/2022.02.03.479055

Cited by 9 publications

(9 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Zhang et al 2022) adapts the optimal transport to constrain the fine-tuned model behaviors, which is a kind of representation regularization. (Xia et al 2022) uses a regularization built on dropout to control the complexity of pre-trained models. Although there are various forms of fine-tuning, it is evident that a gap exists between the learning objectives of the pre-training task and the downstream task.…”

Section: Preliminariesmentioning

confidence: 99%

“…Though the performance is promising, the use of optimal transport requires a high computational cost, thus not applicable for large-scale models. As another example, Xia et al (2022) focuses on molecular graphs, and proposes a new regularization tailored to pre-trained molecular model, but this approach can only be applied to molecular graphs. As a newly developed research direction, prompt-tuning has attracted considerable attention recently.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks

Huang,

Xu,

Jiang

et al. 2024

AAAI

View full text Add to dashboard Cite

The paradigm of pre-training and fine-tuning graph neural networks has attracted wide research attention. In previous studies, the pre-trained models are viewed as universally versatile, and applied for a diverse range of downstream tasks. In many situations, however, this practice results in limited or even negative transfer. This paper, for the first time, emphasizes the specific application scope of graph pre-trained models: not all downstream tasks can effectively benefit from a graph pre-trained model. In light of this, we introduce the measure task consistency to quantify the similarity between graph pre-training and downstream tasks. This measure assesses the extent to which downstream tasks can benefit from specific pre-training tasks. Moreover, a novel fine-tuning strategy, Bridge-Tune, is proposed to further diminish the impact of the difference between pre-training and downstream tasks. The key innovation in Bridge-Tune is an intermediate step that bridges pre-training and downstream tasks. This step takes into account the task differences and further refines the pre-trained model. The superiority of the presented fine-tuning strategy is validated via numerous experiments with different pre-trained models and downstream tasks.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks

Huang,

Xu,

Jiang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…We plot the training and validation curves in Figure 7, from which we can observe that the pre-trained models outperform training from scratch by significant margins. Additionally, for small-scale datasets such as Bace, training from scratch tends to overfit the training data of downstream tasks (Xia et al, 2022d). In contrast, the pre-trained models can mitigate the over-fitting issue.…”

Section: E Implementation Details Of Baselinesmentioning

confidence: 99%

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Xia

Zhao

Hu³

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Recent years have witnessed the prosperity of pre-training graph neural networks (GNNs) for molecules. Typically, atom types as node attributes are randomly masked and GNNs are then trained to predict masked types as in AttrMask \citep{hu2020strategies}, following the Masked Language Modeling (MLM) task of BERT~\citep{devlin2019bert}. However, unlike MLM where the vocabulary is large, the AttrMask pre-training does not learn informative molecular representations due to small and unbalanced atom `vocabulary'. To amend this problem, we propose a variant of VQ-VAE~\citep{van2017neural} as a context-aware tokenizer to encode atom attributes into chemically meaningful discrete codes. This can enlarge the atom vocabulary size and mitigate the quantitative divergence between dominant (e.g., carbons) and rare atoms (e.g., phosphorus). With the enlarged atom `vocabulary', we propose a novel node-level pre-training task, dubbed Masked Atoms Modeling (MAM), to mask some discrete codes randomly and then pre-train GNNs to predict them. MAM also mitigates another issue of AttrMask, namely the negative transfer. It can be easily combined with various pre-training tasks to improve their performance. Furthermore, we propose triplet masked contrastive learning (TMCL) for graph-level pre-training to model the heterogeneous semantic similarity between molecules for effective molecule retrieval. MAM and TMCL constitute a novel pre-training framework, Mole-BERT, which can match or outperform state-of-the-art methods in a fully data-driven manner. We release the code at \textcolor{magenta}{\url{https://github.com/junxia97/Mole-BERT}}.

show abstract

“…To evaluate the transferability of the pre-training scheme, we conduct experiments on transfer learning on molecular property prediction in chemistry and protein function prediction in biology following previous works [16,49,54]. Specifically, we pre-train and finetune the models with different datasets.…”

Section: Transferability (Rq2)mentioning

confidence: 99%

SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation

Xia,

Wu,

Chen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Graph contrastive learning (GCL) has emerged as a dominant technique for graph representation learning which maximizes the mutual information between paired graph augmentations that share the same semantics. Unfortunately, it is difficult to preserve semantics well during augmentations in view of the diverse nature of graph data. Currently, data augmentations in GCL that are designed to preserve semantics broadly fall into three unsatisfactory ways. First, the augmentations can be manually picked per dataset by trial-and-errors. Second, the augmentations can be selected via cumbersome search. Third, the augmentations can be obtained by introducing expensive domain-specific knowledge as guidance. All of these limit the efficiency and more general applicability of existing GCL methods. To circumvent these crucial issues, we propose a Simple framework for GRAph Contrastive lEarning, SimGRACE for brevity, which does not require data augmentations. Specifically, we take original graph as input and GNN model with its perturbed version as two encoders to obtain two correlated views for contrast. SimGRACE is inspired by the observation that graph data can preserve their semantics well during encoder perturbations while not requiring manual trial-and-errors, cumbersome search or expensive domain knowledge for augmentations selection. Also, we explain why SimGRACE can succeed. Furthermore, we devise adversarial training scheme, dubbed AT-SimGRACE, to enhance the robustness of graph contrastive learning and theoretically explain the reasons. Albeit simple, we show that SimGRACE can yield competitive or better performance compared with state-of-the-art methods in terms of generalizability, transferability and robustness, while enjoying unprecedented degree of flexibility and efficiency.

show abstract

Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

Cited by 9 publications

References 18 publications

Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks

Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation

Contact Info

Product

Resources

About