The artificial intelligence chatbot
ChatGPT was used to answer
questions from final exams administered in two general chemistry courses,
including questions with closed-response format and with open-response
format. For closed-response questions, ChatGPT was very capable at
identifying the concept even when the question included a great deal
of chemical symbolism. However, ChatGPT’s success at solving
problems was only 44%, a value well below the class average of 69%.
On open-response questions, ChatGPT’s responses displayed strong
language processing ability with higher performance on questions that
could be solved with more generalizable information compared to questions
that required specific skills, especially when those topics or skills
were primarily found in lecture. Incorrect responses and flawed explanations
were often logically sound and would be persuasive to a novice. The
chatbot is currently ill-equipped to provide reliable answers or explanations
to students for many representative exam questions, but a potential
use is to create assignments in which students analyze and improve
ChatGPT’s responses.