Chemistry
education research demonstrates the value of
open-ended
writing tasks, such as writing-to-learn (WTL) assignments, for supporting
students’ learning with topics including reasoning about reaction
mechanisms. The emergence of generative artificial intelligence (AI)
technology, such as chatbots ChatGPT and Bard, raises concerns regarding
the value of open-ended writing tasks in the classroom; one concern
involves academic integrity and whether students will use these chatbots
to produce sufficient responses to open-ended writing tasks. The present
study investigates the degree to which generative AI chatbots exhibit
mechanistic reasoning in response to organic chemistry WTL assignments.
We produced responses from three generative AI chatbots (ChatGPT-3.5,
ChatGPT-4, and Bard) to two WTL assignments developed to elicit students’
mechanistic reasoning. Using previously reported machine learning
models for analyzing student writing in response to the WTL assignments,
we analyzed the chatbot responses for the inclusion of features pertinent
to mechanistic reasoning. Herein, we report quantitative analyses
of (1) the differences between chatbot responses on the two assignments
and (2) the differences between chatbot and authentic student responses.
Findings indicate that chatbots respond differently to different WTL
assignments. Additionally, the chatbots rarely incorporated the discussion
of electron movement, a key feature of mechanistic reasoning. Furthermore,
the chatbots, in general, do not engage in mechanistic reasoning at
the same level as students. We contextualize the results by considering
academic integrity with the assumption that students’ intentions
are to engage in academically honest behavior, and we focus on understanding
the ethical uses of generative AI for classroom assignments.