ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review

Newton, Philip; Xiromeriti, Maira

doi:10.1080/02602938.2023.2299059

Cited by 13 publications

(6 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authentic extended conversations like these, using the chatbot as an agent to think with have, been called for , but, to our knowledge, have not been reported and are a novel aspect of this investigation. These experiences diverge from early investigations that described the limitations of a chatbot for answering exam questions , and differ from more recent investigations that describe improved exam performance , because they shift the focus from the capability of a chatbot to answer standalone questions to its role in fostering reflection on the part of the user. Unlike earlier investigations, this work included an element of user introspection that highlights the technology’s potential to enrich a teacher’s lesson planning.…”

Section: Discussioncontrasting

confidence: 65%

Using ChatGPT to Support Lesson Planning for the Historical Experiments of Thomson, Millikan, and Rutherford

Clark,

Fhaner,

Stoltzfus

et al. 2024

J. Chem. Educ.

View full text Add to dashboard Cite

Four General Chemistry instructors investigated the use of ChatGPT-4 to improve their lessons plans for the historical experiments of Thomson, Millikan, and Rutherford. The instructors varied in their prior knowledge for these experiments and their initial lessons addressed somewhat different learning objectives. This led to different conversations with the chatbot as the instructors used the resource in different ways and discussed topics they each found relevant. The output from ChatGPT-4 was robust and each instructor identified ways it could be used to improve their instruction. The chatbot was able to accomplish instructional tasks these instructors found useful, such as outlining a lesson plan, recommending resources, discussing instructional strategies, describing calculations, offering explanations for different levels of leaners, and generating assessments. A limitation was its ability create images or visual aids the instructors found useful. Overall, these instructors found the chatbot could support, but not replace, an instructor in a course like General Chemistry.

show abstract

Section: Discussioncontrasting

confidence: 65%

Using ChatGPT to Support Lesson Planning for the Historical Experiments of Thomson, Millikan, and Rutherford

Clark,

Fhaner,

Stoltzfus

et al. 2024

J. Chem. Educ.

View full text Add to dashboard Cite

show abstract

“…Historical data corroborate the extensive nature of academic cheating, with studies revealing a significant proportion of students admitting to such behaviour [16,17]. These concerns around negative trends in academic integrity in online exams were already well advanced when ChatGPT was released, and a new and significant compounding factor was introduced [18][19][20][21]. ChatGPT's ability to answer exam questions has already been demonstrated in multiple studies [21][22][23].…”

Section: Introductionmentioning

confidence: 77%

ChatGPT: The End of Online Exam Integrity?

Susnjak,

McIntosh

2024

Education Sciences

View full text Add to dashboard Cite

This study addresses the significant challenge posed by the use of Large Language Models (LLMs) such as ChatGPT on the integrity of online examinations, focusing on how these models can undermine academic honesty by demonstrating their latent and advanced reasoning capabilities. An iterative self-reflective strategy was developed for invoking critical thinking and higher-order reasoning in LLMs when responding to complex multimodal exam questions involving both visual and textual data. The proposed strategy was demonstrated and evaluated on real exam questions by subject experts and the performance of ChatGPT (GPT-4) with vision was estimated on an additional dataset of 600 text descriptions of multimodal exam questions. The results indicate that the proposed self-reflective strategy can invoke latent multi-hop reasoning capabilities within LLMs, effectively steering them towards correct answers by integrating critical thinking from each modality into the final response. Meanwhile, ChatGPT demonstrated considerable proficiency in being able to answer multimodal exam questions across 12 subjects. These findings challenge prior assertions about the limitations of LLMs in multimodal reasoning and emphasise the need for robust online exam security measures such as advanced proctoring systems and more sophisticated multimodal exam questions to mitigate potential academic misconduct enabled by AI technologies.

show abstract

“… 30 Both studies were conducted using an earlier version of ChatGPT, and more recent iterations are likely to exhibit enhanced performance. 27 , 31 A scoping review by Newton and Xiromeriti 32 revealed that ChatGPT’s performance varied across different evaluation methods and subjects, with ChatGPT 3 passing 20.3% and ChatGPT-4 passing 92.9% of exams. Notably, ChatGPT 3 outperformed human students in 10.9% of exams, while ChatGPT 4 did so in 35%, indicating significant performance improvement in the more advanced version.…”

Section: Discussionmentioning

confidence: 99%

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

Bharatha,

Ojeh,

Fazle Rabbi

et al. 2024

AMEP

View full text Add to dashboard Cite

Introduction This research investigated the capabilities of ChatGPT-4 compared to medical students in answering MCQs using the revised Bloom’s Taxonomy as a benchmark. Methods A cross-sectional study was conducted at The University of the West Indies, Barbados. ChatGPT-4 and medical students were assessed on MCQs from various medical courses using computer-based testing. Results The study included 304 MCQs. Students demonstrated good knowledge, with 78% correctly answering at least 90% of the questions. However, ChatGPT-4 achieved a higher overall score (73.7%) compared to students (66.7%). Course type significantly affected ChatGPT-4’s performance, but revised Bloom’s Taxonomy levels did not. A detailed association check between program levels and Bloom’s taxonomy levels for correct answers by ChatGPT-4 showed a highly significant correlation (p<0.001), reflecting a concentration of “remember-level” questions in preclinical and “evaluate-level” questions in clinical courses. Discussion The study highlights ChatGPT-4’s proficiency in standardized tests but indicates limitations in clinical reasoning and practical skills. This performance discrepancy suggests that the effectiveness of artificial intelligence (AI) varies based on course content. Conclusion While ChatGPT-4 shows promise as an educational tool, its role should be supplementary, with strategic integration into medical education to leverage its strengths and address limitations. Further research is needed to explore AI’s impact on medical education and student performance across educational levels and courses.

show abstract

ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review

Cited by 13 publications

References 42 publications

Using ChatGPT to Support Lesson Planning for the Historical Experiments of Thomson, Millikan, and Rutherford

Using ChatGPT to Support Lesson Planning for the Historical Experiments of Thomson, Millikan, and Rutherford

ChatGPT: The End of Online Exam Integrity?

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

Contact Info

Product

Resources

About