Ontology-Based Generation of Medical, Multi-term MCQs

Abstract

Designing good multiple choice questions (MCQs) for education and assessment is time consuming and error-prone. An abundance of structured and semi-structured data has led to the development of automatic MCQ generation methods. Recently, ontologies have emerged as powerful tools to enable the automatic generation of MCQs. However, current question generation approaches focus on knowledge recall questions. In addition, questions that have so far been generated are, compared to manually created ones, simple and cover only a small subset of the required question complexity space in the education and assessment domain. In this paper, we focus on addressing the limitations of previous approaches by generating questions with complex stems that are suitable for scenarios beyond mere knowledge recall. We present a novel ontology-based approach that exploits classes and existential restrictions to generate case-based questions. Our contribution lies in: (1) the specification of procedure for generating case-based questions which involve (a) assembling complex stems, (b) selecting suitable options, and (c) providing explanations for option correctness/incorrectness, (2) an implementation of the procedure using a medical ontology and (3) and evaluation of our generation technique to test question quality and their suitability in practise. We implement our approach as an application for a medical education scenario on top of a large knowledge base in the medical domain. We generate more than 3 million questions for four physician specialities and evaluate our approach in a user study with 15 medical experts. We find that using a stratified random sample of 435 questions out of which 316 were rated by two experts, 129 (30%) are considered appropriate to be used in exams by both experts and a further 216 (50%) by at least one expert.

A Systematic Review of Automatic Question Generation for Educational Purposes

“…Finally, for ontology-based approaches, similar to the findings reported in the section "Findings of Alsubait's Review", most ontologies used in evaluations were hand-crafted for experimental purposes and the use of real ontologies was neglected, except in Vinu and Kumar (2015b), Leo et al (2019), and Lopetegui et al (2015). Table 9 shows the criteria used for evaluating the quality of questions or their components.…”

“…This aspect of questions has been neglected in early and recent AQG literature. Among the literature that we reviewed, only one study, Leo et al (2019), has generated feedback, alongside the generated questions. They generate feedback as a verbalisation of the axioms used to select options.…”

“…Difficulty control was validated by checking agreement between predicted difficulty and expert prediction in Vinu and Kumar (2015b), Alsubait et al (2016), Seyler et al (2017), Khodeir et al (2018), and Leo et al (2019), by checking agreement between predicted difficulty and student performance in Alsubait et al (2016), Susanti et al (2017a), Lin et al (2015), Wang and Su (2016), Leo et al (2019), and Thomas et al (2019), by employing automatic solvers in Gao et al (2018), or by asking experts to complete a survey after using the tool (Singhal et al 2016). Expert reviews and mock exams are equally represented (seven studies each).…”

“…A dominant approach is the selection of distractors based on their similarity to the key, using different notions of similarity, such as syntax-based similarity (i.e. similar POS, similar letters); Satria and Tokunaga 2017a, b; Jiang and Lee 2017), feature-based similarity(Wita et al 2018;Majumder and Saha 2015; Patra and Saha 2018a, b;Alsubait et al 2016;Leo et al 2019), or contextual similarity(Afzal 2015; Kumar et al 2015a, b;Yaneva and et al 2018;Shah et al 2017;Jiang and Lee 2017). Some studies(Lopetegui et al 2015;Faizan and Lohmann 2018;Faizan et al 2017;…”

The Potential of Open Data to Automatically Create Learning Resources for Smart Learning Environments

“…The main idea of this approach is to define a set of templates that the system applies to the factual knowledge (e.g., vocabulary definitions, relationship among concepts or isolated information about certain details) contained in an ontology, thus creating a vast amount of questions. Many of these systems (e.g., [8]) automatically generate Multiple Choice Questions (MCQs), which are later on used for (self-)evaluation of factual knowledge. Even if these MCQs are automatically created, the production of the ontology remains a problem.…”