Motif-based Graph Self-Supervised Learning for Molecular Property Prediction

Zhang, Zaixi; Liu, Qi; Wang, Hao; Lu, Chengqiang; Lee, Chee‐Kong

doi:10.48550/arxiv.2110.00987

Cited by 6 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SSL was initially applied in computer vision and NLP , that requires large data sets for accurate representation learning. As SSL advances, it has been utilized to predict molecular properties. , For example, it can extract features from unlabeled molecular data. , Likewise, SSL can also extract features from genome data to predict genome function. Recently, many studies have shown that SSL can alleviate the problem of few samples or insufficient supervised information, making it widely applicable in image classification, , recommender systems, protein analysis and design, speech recognition, and other fields.…”

Section: Methods For Small Molecular Data Challengesmentioning

confidence: 99%

“…325,326 For example, it can extract features from unlabeled molecular data. 28,327 Likewise, SSL can also extract features from genome data 328 to predict genome function. Recently, many studies have shown that SSL can alleviate the problem of few samples or insufficient supervised information, making it widely applicable in image classification, 329,330 recommender systems, 331 protein analysis and design, 332 speech recognition, 333 and other fields.…”

Section: Transformersmentioning

confidence: 99%

“…Overall, there are several viable strategies to improve the predictive power of ML or DL models when dealing with small scientific data sets. Commonly used strategies include transfer learning, , combining DL and ML, , GANs, , variational autoencoder (VAE), , self-supervised learning (SSL), , long short-term memory (LSTM), , data augmentation based on physical models, , active learning (AL), , and semi-supervised learning. , However, no paper has provided an organized taxonomy linking these techniques. Therefore, in this review, we conduct a survey on ML or DL prediction using small scientific data sets and aim to create a taxonomy that connects these techniques.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Machine Learning Methods for Small Data Challenges in Molecular Science

et al. 2023

View full text Add to dashboard Cite

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

show abstract

Section: Methods For Small Molecular Data Challengesmentioning

confidence: 99%

Section: Transformersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning Methods for Small Data Challenges in Molecular Science

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Motifs have been proven to benefit many tasks from exploratory analysis to transfer learning. Various algorithms have been proposed to exploit them for contrastive learning (Zhang et al 2020), selfsupervised pretraining (Zhang et al 2021), generation (Jin, Barzilay, and Jaakkola 2020), protein design (Li et al 2022) and drug-drug interaction prediction (Huang et al 2020). But none of them take advantage of motifs to build a heterogeneous graph for molecular property prediction.…”

Section: Related Workmentioning

confidence: 99%

Molformer: Motif-Based Transformer on 3D Heterogeneous Molecular Graphs

Radev

2023

AAAI

View full text Add to dashboard Cite

Procuring expressive molecular representations underpins AI-driven molecule design and scientific discovery. The research mainly focuses on atom-level homogeneous molecular graphs, ignoring the rich information in subgraphs or motifs. However, it has been widely accepted that substructures play a dominant role in identifying and determining molecular properties. To address such issues, we formulate heterogeneous molecular graphs (HMGs) and introduce a novel architecture to exploit both molecular motifs and 3D geometry. Precisely, we extract functional groups as motifs for small molecules and employ reinforcement learning to adaptively select quaternary amino acids as motif candidates for proteins. Then HMGs are constructed with both atom-level and motif-level nodes. To better accommodate those HMGs, we introduce a variant of the Transformer named Molformer, which adopts a heterogeneous self-attention layer to distinguish the interactions between multi-level nodes. Besides, it is also coupled with a multi-scale mechanism to capture fine-grained local patterns with increasing contextual scales. An attentive farthest point sampling algorithm is also proposed to obtain the molecular representations. We validate Molformer across a broad range of domains, including quantum chemistry, physiology, and biophysics. Extensive experiments show that Molformer outperforms or achieves the comparable performance of several state-of-the-art baselines. Our work provides a promising way to utilize informative motifs from the perspective of multi-level graph construction. The code is available at https://github.com/smiles724/Molformer.

show abstract

“…from publicly available database like PubChem or molecular dynamics simulations. In order to utilize the availability of these unlabelled data and overcome the scarcity of labelled data, recently various self-supervised pre-training strategies have been devised for GNNs, and have been successfully demonstrated in social network and biological domains [65][66][67][68][69]. However its applications on quantum mechanical properties have been limited and only available on simple small molecules like those in the QM7 and QM8 datasets [68].…”

mentioning

confidence: 99%

Graph Self-Supervised Learning for Optoelectronic Properties of Organic Semiconductors

Zhang¹,

Liu²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The search for new high-performance organic semiconducting molecules is challenging due to the vastness of the chemical space, machine learning methods, particularly deep learning models like graph neural networks (GNNs), have shown promising potential to address such challenge.However, practical applications of GNNs for chemistry are often limited by the availability of labelled data. Meanwhile, unlabelled molecular data is abundant and could potentially be utilized to alleviate the scarcity issue of labelled data. Here, we advocate the use of self-supervised learning to improve the performance of GNNs by pre-training them with unlabeled molecular data. We investigate regression problems involving ground and excited state properties, both relevant for optoelectronic properties of organic semiconductors. Additionally, we extend the self-supervised learning strategy to molecules in non-equilibrium configurations which are important for studying the effects of disorder. In all cases, we obtain considerable performance improvement over results without pre-training, in particular when labelled training data is limited, and such improvement is attributed to the capability of self-supervised learning in identifying structural similarity among unlabeled molecules.

show abstract

Motif-based Graph Self-Supervised Learning for Molecular Property Prediction

Cited by 6 publications

References 18 publications

Machine Learning Methods for Small Data Challenges in Molecular Science

Machine Learning Methods for Small Data Challenges in Molecular Science

Molformer: Motif-Based Transformer on 3D Heterogeneous Molecular Graphs

Graph Self-Supervised Learning for Optoelectronic Properties of Organic Semiconductors

Contact Info

Product

Resources

About