Transfer learning using attentions across atomic systems with graph neural networks (TAAG)

Abstract

Recent advances in Graph Neural Networks (GNNs) have transformed the space of molecular and catalyst discovery. Even though the underlying physics across these domains remain the same, most prior work has focused on building domain-specific models either in small molecules or in materials. However, building large datasets across domains is computationally expensive, therefore the use of transfer learning (TL) to generalize to different domains is a promising but under-explored approach to this problem. To evaluate this hypothesis, we use a model that is pre-trained on the Open Catalyst Dataset (0C20), and we study the model's behavior when fine-tuned for a set of different datasets and tasks. This includes MD17, CO adsorbate dataset, and across tasks in OC20. Through extensive TL experiments, we demonstrate that initial layers of GNNs learn more basic representation that holds across domains, whereas final layers learn more task-specific features. Moreover, these well-known strategies give significant improvement over the non-pretrained models for in-domain tasks with improvements of 53% and 17% for CO dataset and across OCP task respectively. TL approaches also gives up to 4x speedup in model training depending on the target data and task. However, this does not perform well for MD17 due and gives worse performance than non-pretrained model for few molecules. Based on these observations, we propose TAAG, an attention-based approach that adapts to transfer important features from the interaction layers of GNNs. This method outperforms the best TL approach for MD17 and gives a mean improvement of 6% over the non-pretrained model.

Catlas: an automated framework for catalyst discovery demonstrated for direct syngas conversion

“…For these reasons, direct approaches give reasonable performance for *OH for this use case. The best performing model is a GemNet-dT model that was initially trained to perform iterative relaxation steps, and was finetuned to directly predict the relaxed energy 25 (GemNet-dT FT). This data was used as an initial filtering step.…”

The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts

“…While approaches to fine-tuning vary in which portion of the network's weights are updated, we limit our experiments to updating all the weights and leave more rigorous strategies as future work for the community. 22 For S2EF-Total , we experiment with fine-tuning using different fractions of the OC22 dataset. All fine-tuning experiments are performed using public OC20 adsorptionenergy model checkpoints found at https:// github.com/Open-Catalyst-Project/ocp/ blob/main/MODELS.md.…”

“…However, there are many possible fine-tuning strategies and a large number of variations (e.g. which sections of the GNN to freeze or fit, or leaving this decision to an attention block 22 ), and we expect more progress from the community in this area. These approaches are necessary to encourage the re-use of large models, and to reduce the computational cost of obtaining stateof-the-art models for future small datasets.…”

“…For instance, models can be trained jointly using multiple datasets, or transfer learning may be used to train a model on a larger dataset and fine-tuned on a smaller dataset. Recently, the OC20 dataset enabled the catalysis community to use transfer learning to improve model performance 22 on other smaller datasets. The small molecules and drug discovery communities have seen success in using transfer learning to transfer between varying levels of electronic structure calculations 23 or between related tasks.…”

Open Challenges in Developing Generalizable Large Scale Machine Learning Models for Catalyst Discovery

“…The scale and diversity of OC20 has additionally enabled transfer learning approaches to smaller datasets. Kolluru, et al 46 propose a transfer learning technique to use OC20 pretrained models to improve performance on smaller, out-of-distribution datasets. Similar work has also been demonstrated for other big material datasets.…”