Testing the Ability of Teachers and Students to Differentiate between Essays Generated by ChatGPT and High School Students

Abstract

The release of ChatGPT in late 2022 prompted widespread concern about the implications of artificial intelligence for academic integrity, but thus far there has been little direct empirical evidence to inform this debate. Participants (69 high school teachers, 140 high school students, total N = 209 ) took an AI Identification Test in which they read pairs of essays—one written by a high school student and the other by ChatGPT—and guessed which was generated by the chatbot. Accuracy was only 70% for teachers, and it was slightly worse for students (62%). Self-reported confidence did not predict accuracy, nor did experience with ChatGPT or subject-matter expertise. Well-written student essays were especially hard to differentiate from the ChatGPT texts. In another set of measures, students reported greater optimism than their teachers did about the future role of ChatGPT in education. Students expressed disapproval of submitting ChatGPT-generated essays as one’s own but rated this and other possible academic integrity violations involving ChatGPT less negatively than teachers did. These results form an empirical basis for further work on the relationship between AI and academic integrity.

Can human intelligence safeguard against artificial intelligence? Exploring individual differences in the discernment of human from AI texts

“…25,56 ) or matching of materials to participants' domain expertise (cf. 37,38 ). While greater judgement accuracy for comments may re ect salient stylistic cues present in social media comments, the opportunity to make a side-by-side comparison of the human and AI comments is also likely to have advantaged this component of the task.…”

“…36 Though one study found that high school teachers were slightly better than their students (70% vs. 62%) at determining which in a pair of essays was human vs. AI, self-reported subject-matter expertise in both groups was unrelated to judgment accuracy. 37 However, computer science PhD students were found to be substantially better at detecting AI-generated science abstracts than Wikipedia-type articles written for a lay audience, 38 suggesting that the PhDs' knowledge of stylistic components of scienti c abstracts may have facilitated the distinction for scienti cally styled materials. Likewise, in the aforementioned study exploring empathy and animism, 24 greater prior experience with the tested style of poem (Haiku) was associated with stronger judgment accuracy.…”

A Comparative Analysis of the Rating of College Students’ Essays by ChatGPT versus Human Raters

“…Hence, academic institutions need to revisit and update their policies on students' and teachers' conduct in the use of generative AI tools. Teacher training and assessment practices in writing courses need to be upgraded to include academic integrity issues, such as plagiarism, originality issues, assignments, and online and home-based exams, factors which can be compromised when students use generative AI tools (Imran & Almusharraf, 2023;Waltzer et al, 2023).…”

The Effect of Teachers Beliefs and Experiences on the Use of ChatGPT in STEM Disciplines

“…A prevalent issue in employing ChatGPT is the ease with which the AI can be utilized to generate texts, posing challenges in detecting plagiarized content. A study published in June 2023 assessed educators' and students' ability to differentiate between machine-generated and human-produced texts (Waltzer et al, 2023). While educators could accurately attribute texts to either the AI or students with a 70% accuracy rate, students achieved a 62% accuracy rate (Waltzer et al, 2023).…”

“…A study published in June 2023 assessed educators' and students' ability to differentiate between machine-generated and human-produced texts (Waltzer et al, 2023). While educators could accurately attribute texts to either the AI or students with a 70% accuracy rate, students achieved a 62% accuracy rate (Waltzer et al, 2023). With the continuous enhancement of the GPT model, it is anticipated that human detection of plagiarism will become increasingly challenging.…”