Do large language models know chemistry?

Abstract

Mostly yes. We systematically evaluate machine learning large language models (LLMs) that generate code in the context of chemistry. We produce a benchmark set of problems, and evaluate these models based on correctness of code by automated testing and evaluation by experts. We find recent LLMs are able to write correct code across a variety of topics in chemistry and their accuracy can be increased by 30 percentage points via prompt engineering strategies, like putting copyright notices at the top of files. These dataset and evaluation tools are open source which can be contributed to or built upon by future researchers, and will serve as a community resource for evaluating the performance of new models as they emerge. We also describe some good practices for employing LLMs in chemistry. The general success of these models demonstrates that their impact on chemistry teaching and research is poised to be enormous.

Formalizing chemical physics using the Lean theorem prover

“…AI tools that can autoformalize the informal scientic literature, generate novel theories, and auto-complete complex proofs could open new avenues for automating theory discovery. LLMs have demonstrated capabilities in solving chemistry problems, 90,91 as well as answering scientic question-and-answer problems invoking quantitative reasoning. 92 However, LLMs are unreliable they famously "hallucinate" (generate falsehoods) and are biased or unreliable evaluators of their own outputs.…”

Optimizing Learning Across Multimodal Transfer Features for Modelling Olfactory Perception

“…It works well for the defined downstream predictive task, especially demonstrating improved performance in small data settings. The question of whether large language models, such as GPT-3, trained on non-chemical corpora, can acquire meaningful knowledge in the field of chemistry has also been investigated in a recent study 40 .…”

Interactive Quantum Chemistry Enabled by Machine Learning, Graphical Processing Units, and Cloud Computing

“…NLP algorithms have exciting potential for chemistry applications; for example, voiceactivated, hands-free software could be used to read and write electronic notebooks while performing experiments. In computational chemistry, vocal prompts could be used to input the commands or as support for programming tasks; Hocky, White, and colleagues (93,94) recently showed that Codex (95) can generate code for chemical applications from natural language prompts. Further applications of NLP in chemistry are certainly promising.…”