Adaptive Transfer Learning on Graph Neural Networks

Han, Xueting; Huang, Zhenhuan; An, Bang; Bai, Jing

doi:10.1145/3447548.3467450

Cited by 31 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analogously, MPG [Li et al ., 2021b] learns to compare two half-graphs (each decomposed from a graph sample) and discriminate whether they come from the same source. Despite the progress in molecular graph pre-training, few efforts have been devoted to the fine-tuning except for a recent work [Han and others., 2021] that adaptively selects and combines various auxiliary tasks with the target task in the fine-tuning stage to improve performance, which is impractical because auxiliary tasks are often unavailable during fine-tuning.…”

Section: Related Workmentioning

confidence: 99%

Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

Xia

Zheng

Tan

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) and Transformer have emerged as dominant tools for AI-driven drug discovery. Many state-of-the-art methods first pre-train GNNs or the hybrid of GNNs and Transformer on a large molecular database and then fine-tune on downstream tasks. However, different from other domains such as computer vision (CV) or natural language processing (NLP), getting labels for molecular data of downstream tasks often requires resource-intensive wet-lab experiments. Besides, the pre-trained models are often of extremely high complexity with huge parameters. These often cause the fine-tuned model to over-fit the training data of downstream tasks and significantly deteriorate the performance. To alleviate these critical yet underexplored issues, we propose two straightforward yet effective strategies to attain better generalization performance: 1. MolAug, which enriches the molecular datasets of downstream tasks with chemical homologies and enantiomers; 2. WordReg, which controls the complexity of the pre-trained models with a smoothness-inducing regularization built on dropout. Extensive experiments demonstrate that our proposed strategies achieve notable and consistent improvements over vanilla fine-tuning and yield multiple state-of-the-art results. Also, these strategies are model-agnostic and readily pluggable into fine-tuning of various pre-trained molecular graph models. We will release the code and the fine-tuned models.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

Xia

Zheng

Tan

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Fine-tuning is a dominant technique to adapt the knowledge to various downstream tasks, whereas it is confronted with the issue of catastrophic forgetting [52], which means the MPMs often forget their learned knowledge during fine-tuning. To alleviate this issue, Han et al [30] adaptively select and combine various pretraining tasks along with the target tasks in the fine-tuning stage to achieve a better adaptation. This strategy preserves sufficient knowledge captured by self-supervised pre-training tasks and improves the effectiveness of transfer learning on molecular pre-training.…”

Section: Towards Better Knowledge Transfermentioning

confidence: 99%

A Systematic Survey of Molecular Pre-trained Models

Xia¹,

Zhu²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Obtaining effective molecular representations is at the core of a series of important chemical tasks ranging from property prediction to drug design. So far, deep learning has achieved remarkable success in learning representations for molecules through automated feature learning in a data-driven fashion. However, training deep neural networks from scratch often requires sufficient labeled molecules which are expensive to acquire in real-world scenarios. To alleviate this issue, inspired by the success of the pretrain-then-finetune paradigm in natural language processing, tremendous efforts have been devoted to Molecular Pre-trained Models (MPMs), where neural networks are pre-trained using large-scale unlabeled molecular databases and then fine-tuned for diverse downstream tasks. Despite the prosperity, this field is fast-growing and a systematic roadmap is urgently needed for both methodology advancements and practical applications in both machine learning and scientific communities. To this end, this paper provides a systematic survey of pre-trained models for molecular representations. Firstly, to motivate MPMs studies, we highlight the limitations of training deep neural networks for molecular representations. Next, we systematically review recent advances on this topic from several key perspectives including molecular descriptors, encoder architectures, pre-training strategies, and applications. Finally, we identify several challenges and discuss promising future research directions.

show abstract

“…Our Progressive Network configuration facilitates the search to find from which layers we retrain the network. In credit risk, where the data labels are scarce, the approach such as transferring learned knowledge from self-supervised tasks to downstream tasks could improve the performance of the network (Han et al, 2021 ).…”

Section: Related Studiesmentioning

confidence: 99%

Credit Risk Modeling Using Transfer Learning and Domain Adaptation

Suryanto¹,

Mahidadia²,

Bain³

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

In the domain of credit risk assessment lenders may have limited or no data on the historical lending outcomes of credit applicants. Typically this disproportionately affects Micro, Small, and Medium Enterprises (MSMEs), for which credit may be restricted or too costly, due to the difficulty of predicting the Probability of Default (PD). However, if data from other related credit risk domains is available Transfer Learning may be applied to successfully train models, e.g., from the credit card lending and debt consolidation (CD) domains to predict in the small business lending domain. In this article, we report successful results from an approach using transfer learning to predict the probability of default based on the novel concept of Progressive Shift Contribution (PSC) from source to target domain. Toward real-world application by lenders of this approach, we further address two key questions. The first is to explain transfer learning models, and the second is to adjust features when the source and target domains differ. To address the first question, we apply Shapley values to investigate how and why transfer learning improves model accuracy, and also propose and test a domain adaptation approach to address the second. These results show that adaptation improves model accuracy in addition to the improvement from transfer learning. We extend this by proposing and testing a combined strategy of feature selection and adaptation to convert values of source domain features to better approximate values of target domain features. Our approach includes a strategy to choose features for adaptation and an algorithm to adapt the values of these features. In this setting, transfer learning appears to improve model accuracy by increasing the contribution of less predictive features. Although the percentage improvements are small, such improvements in real world lending could be of significant economic importance.

show abstract

Adaptive Transfer Learning on Graph Neural Networks

Cited by 31 publications

References 19 publications

Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models

A Systematic Survey of Molecular Pre-trained Models

Credit Risk Modeling Using Transfer Learning and Domain Adaptation

Contact Info

Product

Resources

About