On ChatGPT: what promise remains for multiple choice assessment?

Abstract

Multiple-choice quizzes (MCQs) are a popular form of assessment. A rapid shift to online assessment during the Covid-19 pandemic in 2020, drove the uptake of MCQs, yet limited invigilation and wide access to material on the internet allow students to solve the questions via internet search. ChatGPT, an artificial intelligence (AI) agent trained on a large language model, exacerbates this challenge as it responds to information retrieval questions with speed and a good level of accuracy. In this opinion piece, I contend that while the place of MCQ in summative assessment may be uncertain, current shortcomings of ChatGPT offer opportunities for continued formative use. I outline how ChatGPT’s limitations can inform effective question design. I provide tips for effective multiple-choice question design and outline implications for both academics and learning developers. This piece contributes to emerging debate on the impact of artificial intelligence on assessment in higher education. Its purpose is threefold: to (1) enhance academics’ understanding of effective MCQ design, (2) promote shared understanding and inform dialogue between academics and learning developers about MCQ assessment, and (3) highlight the potential implications on learning support.

Below average ChatGPT performance in medical microbiology exam compared to university students

“…Lastly, the study opens the door to a broader inquiry into the validity and reliability of MCQ-based assessments in higher education since passing these exams can be achieved by AI-models. Thus, refined approaches for effective design of MCQs is needed to maintain the reliability of MCQs as an assessment method in higher education (Gonsalves, 2023). Future studies are recommended taken into consideration the issues of rigorous design, variable tested subjects, different language and cultural aspects, and different exam settings.…”

Creating subject competence translation tests with GPT-4: A case study in English-to-Turkish translations in the engineering domain

“…The limitations of current language models can shed light for designing MCQs that promote active engagement and critical thinking. Strategies like incorporating multimedia, using conditional logic, grounding questions in current events, and meticulously crafting distractors can counteract the model's pattern recognition, making it more di cult for the model to simply identify answer choices based on statistical patterns (Gonsalves, 2023).…”

Generative artificial intelligence (AI) in higher education: a comprehensive review of challenges, opportunities, and implications

“…The literature identifies multiple solutions to the above problem. Gonsalves (2023) suggests that educators using multiple choice test for assessment may consider outsmarting the system by taking advantage of the current limitations of generative AI platforms, such as their inability to interpret visual media or understand up-to-date information beyond its training data. For instance, multiple choice questions could incorporate visual elements, such as images, figures, or charts, and require students to interact with these elements.…”