Pre-training via Denoising for Molecular Property Prediction

Zaidi, Sheheryar; Schaarschmidt, Michael; Martens, J.; Kim, Hyunjik; Teh, Yee Whye; Sánchez‐González, Álvaro; Battaglia, Peter W.; Pascanu, Razvan; Godwin, Jonathan

doi:10.48550/arxiv.2206.00133

Cited by 16 publications

(25 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first one is a supervised learning objective, which aims to predict the HOMO-LUMO energy gap of each molecule. Besides, we also use a self-supervised learning objective called 3D Position Denoising Zaidi et al, 2022), which is particularly effective. During training, if a data instance is in the 3D mode, we add Gaussian noise to the position of each atom and require the model to predict the noise from the noisy input.…”

Section: Large-scale Pre-trainingmentioning

confidence: 99%

See 1 more Smart Citation

One Transformer Can Understand Both 2D & 3D Molecular Data

Luo¹,

Chen²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unlike vision and language data which usually has a unique format, molecules can naturally be characterized using different chemical formulations. One can view a molecule as a 2D graph or define it as a collection of atoms located in a 3D space. For molecular representation learning, most previous works designed neural networks only for a particular data format, making the learned models likely to fail for other data formats. We believe a general-purpose neural network model for chemistry should be able to handle molecular tasks across data modalities. To achieve this goal, in this work, we develop a novel Transformer-based Molecular model called Transformer-M, which can take molecular data of 2D or 3D formats as input and generate meaningful semantic representations. Using the standard Transformer as the backbone architecture, Transformer-M develops two separated channels to encode 2D and 3D structural information and incorporate them with the atom features in the network modules. When the input data is in a particular format, the corresponding channel will be activated, and the other will be disabled. By training on 2D and 3D molecular data with properly designed supervised signals, Transformer-M automatically learns to leverage knowledge from different data modalities and correctly capture the representations. We conducted extensive experiments for Transformer-M. All empirical results show that Transformer-M can simultaneously achieve strong performance on 2D and 3D tasks, suggesting its broad applicability. The code and models will be made publicly available at https://github.com/lsj2408/Transformer-M.

show abstract

Section: Large-scale Pre-trainingmentioning

confidence: 99%

“…Prediction head for position output. (3D Position Denoising) Following ; Zaidi et al (2022), we further adopt the 3D Position Denoising task as a self-supervised learning objective. During training, if a data instance is in the 3D mode, we add Gaussian noise to the positions of each atom.…”

Section: A Implementation Details Of Transformer-mmentioning

confidence: 99%

One Transformer Can Understand Both 2D & 3D Molecular Data

Luo¹,

Chen²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Inspired by Noisy Nodes [26] that incorporates denoising (Fig. 3g) as an auxiliary task to improve performance, a recent work [128] adds noise to atomic coordinates of 3D geometry and pre-trains the encoders to predict the noise. They further demonstrate that denoising objective in molecular pre-training is approximately equivalent to learning a molecular force field, shedding light on how denoising aids molecular pre-training.…”

Section: Denoising Modeling (Dm)mentioning

confidence: 99%

“…You et al [125] propose to maximize the agreements between paired molecular graph augmentations using a contrastive objective [9]. Zaidi et al [128] demonstrate that performing denoising on the conformational space can be helpful for learning molecular force fields.…”

Section: Introductionmentioning

confidence: 99%

A Systematic Survey of Molecular Pre-trained Models

Xia¹,

Zhu²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Obtaining effective molecular representations is at the core of a series of important chemical tasks ranging from property prediction to drug design. So far, deep learning has achieved remarkable success in learning representations for molecules through automated feature learning in a data-driven fashion. However, training deep neural networks from scratch often requires sufficient labeled molecules which are expensive to acquire in real-world scenarios. To alleviate this issue, inspired by the success of the pretrain-then-finetune paradigm in natural language processing, tremendous efforts have been devoted to Molecular Pre-trained Models (MPMs), where neural networks are pre-trained using large-scale unlabeled molecular databases and then fine-tuned for diverse downstream tasks. Despite the prosperity, this field is fast-growing and a systematic roadmap is urgently needed for both methodology advancements and practical applications in both machine learning and scientific communities. To this end, this paper provides a systematic survey of pre-trained models for molecular representations. Firstly, to motivate MPMs studies, we highlight the limitations of training deep neural networks for molecular representations. Next, we systematically review recent advances on this topic from several key perspectives including molecular descriptors, encoder architectures, pre-training strategies, and applications. Finally, we identify several challenges and discuss promising future research directions.

show abstract

“…This key observation underpins the significant recent advances in generative diffusion models, which use an estimate of the score-function to generate samples (Ho et al, 2020;Song et al, 2021). The recent success of DAEs in generative modelling has not yet translated to representation learning, with some exceptions (Asiedu et al, 2022;Zaidi et al, 2022). In this work we exploit a denoising autoencoder to eliminate the MAE inefficiency of reconstructing unmasked patches but never using them.…”

Section: Related Workmentioning

confidence: 99%

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Mishra¹,

Robinson²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the high-frequency components of an image. The combined approach results in a robust, scalable and simple-to-implement algorithm. The training process is symmetric, with 50% of patches in both views being masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies demonstrate that CAN achieves strong downstream performance under both linear and finetuning evaluations on transfer learning and robustness tasks. CAN outperforms MAE and SimCLR when pre-training on ImageNet, but is especially useful for pre-training on larger uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN achieves 75.4% compared to 73.4% for SimCLR and 64.1% for MAE. The finetuned performance on ImageNet of our ViT-L model is 86.1%, compared to 85.5% for SimCLR, and 85.4% for MAE. The overall FLOPs load of SimCLR is 70% higher than CAN for ViT-L models 1 . * Equal contribution, order chosen randomly. Work done during an internship at Google. 1 Code will be released.

show abstract

Pre-training via Denoising for Molecular Property Prediction

Cited by 16 publications

References 19 publications

One Transformer Can Understand Both 2D & 3D Molecular Data

One Transformer Can Understand Both 2D & 3D Molecular Data

A Systematic Survey of Molecular Pre-trained Models

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Contact Info

Product

Resources

About