Controlling item difficulty for automatic vocabulary question generation

Abstract

The present study investigates the best factor for controlling the item difficulty of multiple-choice English vocabulary questions generated by an automatic question generation system. Three factors are considered for controlling item difficulty: (1) reading passage difficulty, (2) semantic similarity between the correct answer and distractors, and (3) the distractor word difficulty level. An experiment was conducted by administering machine-generated items to three groups of English learners. The groups were determined based on their standardised English test scores. In total, 120 items, generated using combinations of the above three factors, were tested. The results reveal that the distractor word difficulty level had the greatest impact on item difficulty, but this tendency changed depending on the proficiency of the test takers. These results will be of use when implementing a fully automatic system for administrating tests.

A Systematic Review of Automatic Question Generation for Educational Purposes

“…Despite the growth in AQG, only 14 studies have dealt with difficulty. Eight of these studies focus on the difficulty of questions belonging to a particular domain, such as mathematical word problems (Wang and Su 2016;Khodeir et al 2018), geometry questions (Singhal et al 2016), vocabulary questions (Susanti et al 2017a), reading comprehension questions (Gao et al 2018), DFA problems (Shenoy et al 2016), code-tracing questions (Thomas et al 2019), and medical case-based questions Kurdi et al 2019). The remaining six focus on controlling the difficulty of non-domain-specific questions (Lin et al 2015;Alsubait et al 2016;Kurdi et al 2017;Faizan and Lohmann 2018;Faizan et al 2017;Seyler et al 2017;Kumar 2015a, 2017a;Vinu et al 2016;Kumar 2017b, 2015b).…”

“…Difficulty control was validated by checking agreement between predicted difficulty and expert prediction in Vinu and Kumar (2015b), Alsubait et al (2016), Seyler et al (2017), Khodeir et al (2018), and Leo et al (2019), by checking agreement between predicted difficulty and student performance in Alsubait et al (2016), Susanti et al (2017a), Lin et al (2015), Wang and Su (2016), Leo et al (2019), and Thomas et al (2019), by employing automatic solvers in Gao et al (2018), or by asking experts to complete a survey after using the tool (Singhal et al 2016). Expert reviews and mock exams are equally represented (seven studies each).…”

“…On the other hand, grammatical inconsistency between the stem and the correct option can confuse test takers who have the required knowledge and would have been likely to select the key otherwise. Providing different phrasing for the question text is also of importance, playing a role in keeping test Table 6 Features proposed for controlling the difficulty of generated questions Reference Feature Lin et al (2015) Feature-based similarity between key and distractors Singhal et al (2015aSinghal et al ( , b, 2016 Number and type of domain-objects involved Number and type of domain-rules involved User given scenarios Length of the solution Direct/indirect use of rules involved Susanti et al (2017aSusanti et al ( , b, 2015Susanti et al ( , 2016 Reading passage difficulty Contextual similarity between key and distractors Distractor word difficulty level Kumar (2015a, 2017a), Quality of hints (i.e. how much they reduce the answer space) takers engaged.…”

“…#Q = no of evaluated questions; #P = no of participants (whether S = student(s), E = expert(s), C = co-worker(s) or A = author(s)); avg. = average; NR = not reported in the paper; NC = not clear; NA = not applicable; * = not reported but calculated based on provided data; and (+) = refer to the paper for extra information about the results or the context of the study Reference International Journal of Artificial Intelligence in Education (2020) 30:121-204 Khodeir et al (2018Khodeir et al ( , 2018 math 25 4 E 82% questions were thought to be human-authored 3 categories (system-generated; human-generated; unsure) Susanti et al (2015Susanti et al ( , 2016Susanti et al ( , 2017a language 22 7 E 45% questions were thought to be human-authored binary choice (human-generate; machine-generated) language 69 364 C 67% questions were thought to be human-authored Overlap with human generated questions Shirude et al (2015) generic NR 12 E 63.15% types of questions generated by the human are covered by the generator NR Vinu and Kumar (2015a generic NR NR recall = 43% to 81%, precision = 72% to 93% (+) Liu et al (2017) language (RC) 600 3 E recall = 64%, precision = 69% (+) NR Jouault et al (2015aJouault et al ( , b, 2016aJouault et al ( , b, 2017 history 69 1 E 84% of the human-authored questions are covered by auto-generated questions coverages means that both questions cover the same knowledge…”

Maintaining reading experience continuity across e-book revisions

“…Cosine similarity is a term-based similarity measure baseline of similarity between two vectors of an inner product space that measures the cosine of the angle between them (Gomaa and Fahmy 2013). It has been widely used in several text semantic analysis tasks in Landauer and Dumais (1997); Mihalcea et al (2006); Cheng et al (2008); Susanti et al (2017). Jaccard similarity coefficient (Roussinov and Zhao 2003) is a statistical measure of the extent of overlapping between two vectors.…”