Comparing the Performance of College Chemistry Students with ChatGPT for Calculations Involving Acids and Bases

Abstract

Student performance on open-response calculations involving acid and base solutions before and after instruction in general chemistry and analytical chemistry courses was compared with the output from the artificial intelligence chatbot ChatGPT. Applying a theoretical model of expertise for problem solving that includes problem conceptualization, problem strategy, and solution, it is found that students errors following instruction primarily involved problem conceptualization and the misapplication of heuristics like the Henderson−Hasselbalch equation When the same problems were used as input to ChatGPT the responses were comparable to worked examples found in general chemistry textbooks in terms of length and detail and usually displayed strong problem conceptualization. Response accuracy of the chatbot varied greatly for different topics, being best for calculations of pH for a strong acid or strong base and much lower for more complex problems involving titrations or aqueous salts. Chatbot and student errors differed in that the chatbot did not misapply heuristics but did make mathematical errors uncommon for students. The variability in the correctness of ChatGPT's responses and the nature of its errors vis-a-vis students will influence its potential use as an instructional resource for calculations involving acids and bases.

Students’ Experience of a ChatGPT Enabled Final Exam in a Non-Majors Chemistry Course

“…Investigations have considered ChatGPT performance on exam questions from courses such as general chemistry and pharmaceutical science . More specific content investigations with a focus on acid/base calculations and organic chemistry have also been reported. Notably critical evaluations of ChatGPT performance on chemistry tasks have also resulted in these broad efforts to probe content accuracy. , …”

Comment on “Comparing the Performance of College Chemistry Students with ChatGPT for Calculations Involving Acids and Bases”

“…A complete computational notebook containing all necessary code and results is provided online. 3 In principle, similar results could be obtained following Clark et al's process of typing the entries into the web browserbased ChatGPT Plus; 4 programming merely avoids tedious data entry.…”

“…The problems and errors of LLM outputs discussed by Clark et al are only true for their specific choice of GPT-3.5; these limitations are mostly resolved by using a better model and entirely resolved by using an improved prompt and calculator tool. Teachers should assume that students already have access to this functionality; eventually, it will be the default.…”

“…Recently in this Journal , Clark et al identified deficiencies in the performance of the GPT-3.5 large language model (LLM) on undergraduate acid–base calculation problems that limit its educational applications . To investigate whether their critiques remain relevant to the GPT-4 model (released November 2023), Mathematica 14.0 was used to programmatically access the model and define prompts and tools ( vide infra ) and applied to the two most difficult questions reported by Clark et al for the purpose of illustrationpH calculations of a salt solution and following titration of a weak base by a strong acid.…”

ChatGPT Convincingly Explains Organic Chemistry Reaction Mechanisms Slightly Inaccurately with High Levels of Explanation Sophistication

“…Clark also asked ChatGPT-3.5 to solve a set of 15 multiple-choice general chemistry cumulative final exam questions and found that the chatbot was able to answer 40% of the questions correctly contrasting with the 76% student average . Clark and colleagues conducted a follow-up study specifically on questions involving acid and base calculations . Generally, ChatGPT-3.5 could calculate pH involving strong acids/bases and volume for neutralization well but performed weaker for problems involving weak acids/bases and aqueous salts and could not calculate pH for titration questions.…”