Prompt engineering of GPT-4 for chemical research: what can/cannot be done?

Abstract

This paper evaluates the capabilities and limitations of the Generative Pre-trained Transformer 4 (GPT-4) in chemical research. Although GPT-4 exhibits remarkable proficiencies, it is evident that the quality of input data significantly affects its performance. We explore GPT-4's potential in chemical tasks, such as foundational chemistry knowledge, cheminformatics, data analysis, problem prediction, and proposal abilities. While the language model partially outperformed traditional methods, such as black-box optimization, it fell short against specialized algorithms, highlighting the need for their combined use. The paper shares the prompts given to GPT-4 and its responses, providing a resource for prompt engineering within the community, and concludes with a discussion on the future of chemical research using large language models.

Shaping the Water-Harvesting Behavior of Metal–Organic Frameworks Aided by Fine-Tuned GPT Models

“…Simply put, output representations should be valid chemical structures, respecting all rules of valency and bonding, and should accurately reflect any assigned mutations or modifications from the original structure. While LLMs like GPT-3.5, GPT-4, and their chatbot adaptations, known as “ChatGPT”, offer the advantage of interpreting human instructions in a conversational format, which makes it simpler to convey abstract mutations and modifications, the early performance evaluations of these models have shown their limitations. − Despite demonstrating certain levels of understanding of the underlying syntax and chemistry, these models sometimes suffer from “hallucinations” in their generated SMILES strings, which appear correct in formatting but are either chemically invalid or slightly misaligned when closely examined.…”

“…The objective is to provide various examples to teach the model the intricacies behind such edits. In this case, this approach is more powerful than the zero-shot or few-shot prompt engineering strategies, ,− as the MOF linker mutation methods and actions are diverse, and the principles behind molecular editing encompass not only chemical rules like bonding and valency but also the construction of syntactically correct text-based chemical representations (Figures S3–S5). Fine-tuning through API, in comparison to in-context learning in prompt engineering, is not constrained by token limits, allowing the training of models on a substantial volume of examples and case studies (Table S2 and Figure S6).…”

Artificial Intelligence Chatbots in Chemical Information Seeking: Educational Insights through a SWOT analysis

“…Chemistry education and teacher education will benefit from AI chatbots similarly to any other domain. Learners can refine information to knowledge via learning discussions, check facts, and prompt definitions for concepts (Bawden and Robinson 2012;Hatakeyama-Sato et al 2023). However, one must be aware of the limitations of LLMs and analyze or triangulate the generated information before using it.…”

Comment on “Comparing the Performance of College Chemistry Students with ChatGPT for Calculations Involving Acids and Bases”

“…GPT-4 has better performance than GPT-3.5 in a variety of tasks, including chemistry; therefore, one expects similar improvements for these acid–base problems. The same initial prompt was provided for both questions, written to facilitate automatic grading: .…”

“…Alternatively, the ability to generate reliable , human-readable explanations on demand means AI is more generally applicable than Clark et al suggest. Third, there is little doubt that AI in general, , and LLMs in particular, , provide a powerful new tool for chemistry, so teaching its effective use is imperative. The procedure described hereproviding tools and structured reasoning promptsis already being used by ChemCrow and AI-Coscientist to direct chemical research.…”