Accurate, interpretable predictions of materials properties within transformer language models

Korolev, Vadim V.; Проценко, П.

doi:10.1016/j.patter.2023.100803

Cited by 15 publications

(3 citation statements)

References 123 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The lack of data for the end point of interest can be addressed by leveraging advanced techniques, including transfer learning − and self-supervised learning. , To assess the potential usefulness of applying CG 2 -NNs in conjunction with one of such algorithms, we consider the task of band gap prediction across different classes of reticular materials: the models pretrained on the MOF band gap are used to evaluate the same property in COFs (Figure a, Table S3). The models trained from scratch, i.e., without pretraining, serve as a baseline.…”

Section: Resultsmentioning

confidence: 99%

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Korolev,

Mitrofanov

2024

J. Chem. Inf. Model.

Self Cite

View full text Add to dashboard Cite

Reticular materials, including metal−organic frameworks and covalent organic frameworks, combine the relative ease of synthesis and an impressive range of applications in various fields from gas storage to biomedicine. Diverse properties arise from the variation of building units�metal centers and organic linkers�in almost infinite chemical space. Such variation substantially complicates the experimental design and promotes the use of computational methods. In particular, the most successful artificial intelligence algorithms for predicting the properties of reticular materials are atomic-level graph neural networks, which optionally incorporate domain knowledge. Nonetheless, the data-driven inverse design involving these models suffers from the incorporation of irrelevant and redundant features such as a full atomistic graph and network topology. In this study, we propose a new way of representing materials, aiming to overcome the limitations of existing methods; the message passing is performed on a coarse-grained crystal graph that comprises molecular building units. To highlight the merits of our approach, we assessed the predictive performance and energy efficiency of neural networks built on different materials representations, including composition-based and crystal-structure-aware models. Coarse-grained crystal graph neural networks showed decent accuracy at low computational costs, making them a valuable alternative to omnipresent atomic-level algorithms. Moreover, the presented models can be successfully integrated into an inverse materials design pipeline as estimators of the objective function. Overall, the coarse-grained crystal graph framework is aimed at challenging the prevailing atom-centric perspective on reticular materials design.

show abstract

Section: Resultsmentioning

confidence: 99%

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Korolev,

Mitrofanov

2024

J. Chem. Inf. Model.

Self Cite

View full text Add to dashboard Cite

show abstract

“…They are trained on a large amount of data with multiple modalities of concepts. They have been evolving interactively with large language models (LLMs) [19,20], which have also brought a significant impact on materials science [21][22][23][24][25][26][27][28][29]. FMs and LLMs often refer the same as natural language is so common for us, which should also be the case for materials science; however, FMs more specifically connect different modalities and are expected to open up ways of conceptual representation.…”

Section: Introductionmentioning

confidence: 99%

Graph-Text Contrastive Learning of Inorganic Crystal Structure toward a Foundation Model of Inorganic Materials

Ozawa,

Suzuki,

Tonogai

et al. 2024

Preprint

View full text Add to dashboard Cite

Developing foundation models for materials science has attracted attention. However, there is a lack of work on inorganic materials due to the difficulty in the comprehensive representation of geometric concepts composing crystals, from the local atomic environments, their connections, and the global symmetries. We present a contrastive learning of inorganic crystal structure (CLICS) for embedding the geometric concepts, which contrasts texts representing the contextual patterns of geometries with the crystal graphs. We demonstrate that the geometric concepts are integrally embedded on CLICS feature space, through experiments of concept retrieval from crystal graphs, similar structure search, and few-shot/imbalanced crystal structure classification.

show abstract

“…There have been several previous research works for forward prediction of material properties, such as descriptor-based methods, including models in MatMiner 14 and classical forcefield inspired descriptors (CFID); 15 graph-based methods, such as crystal graph convolutional neural networks (CGCNNs), 16 Materials Graph Network (MEGNet), 17 Atomistic Line Graph Neural Network (ALIGNN), 18 CoGN, 19 M3GNet, 20 Matformer, 21 and ComFormer; 22 and language-based methods, such as LLM-Prop 23 etc. Similarly, some of the earlier inverse design methods are Crystal Diffusional Variational Autoencoder (CDVAE), 24 Fourier Transformed Crystal Properties (FTCP), 25 G-SchNet, 26 MatBERT, 27 CrystaLLM, 28 Crystal-LLM, 29 and xyztransformer. 30 AtomGPT is a deep learning model 6 that leverages a few transformer architectures such as GPT2 31 and quantized Mistral-AI 32 to learn the complex relationships between atomic structures and material properties from datasets such as JARVIS-DFT.…”

mentioning

confidence: 99%

AtomGPT: Atomistic Generative Pretrained Transformer for Forward and Inverse Materials Design

Choudhary

2024

J. Phys. Chem. Lett.

View full text Add to dashboard Cite

Large language models (LLMs) such as generative pretrained transformers (GPTs) have shown potential for various commercial applications, but their applicability for materials design remains underexplored. In this Letter, AtomGPT is introduced as a model specifically developed for materials design based on transformer architectures, demonstrating capabilities for both atomistic property prediction and structure generation. This study shows that a combination of chemical and structural text descriptions can efficiently predict material properties with accuracy comparable to graph neural network models, including formation energies, electronic bandgaps from two different methods, and superconducting transition temperatures. Furthermore, AtomGPT can generate atomic structures for tasks such as designing new superconductors, with the predictions validated through density functional theory calculations. This work paves the way for leveraging LLMs in forward and inverse materials design, offering an efficient approach to the discovery and optimization of materials.

show abstract

Accurate, interpretable predictions of materials properties within transformer language models

Cited by 15 publications

References 123 publications

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Graph-Text Contrastive Learning of Inorganic Crystal Structure toward a Foundation Model of Inorganic Materials

AtomGPT: Atomistic Generative Pretrained Transformer for Forward and Inverse Materials Design

Contact Info

Product

Resources

About