Evaluation of automatically generated English vocabulary questions

Susanti, Yuni; Tokunaga, Takenobu; Nishikawa, Hitoshi; Obari, Hiroyuki

doi:10.1186/s41039-017-0051-y

Cited by 15 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different qualitative and quantitative analyses were carried out to evaluate questions auto-generated from domain ontologies (Alsubait et al 2014;Vinu and Kumar 2017;Seyler et al 2016;Susanti et al 2017). Papasalouros et al (2017;2011) autogenerated multiple choice questions (MCQs) from the Eupalineio Tunnel ontology, which is a domain ontology about the ancient Greek history.…”

Section: Related Workmentioning

confidence: 99%

Evaluating the quality of the ontology-based auto-generated questions

Demaidi

Gaber

Filer

2017

Smart Learn. Environ.

View full text Add to dashboard Cite

An ontology is a knowledge representation structure which has been used in Virtual Learning Environments (VLEs) to describe educational courses by capturing the concepts and the relationships between them. Several ontology-based question generators used ontologies to auto-generate questions, which aimed to assess students' at different levels in Bloom's taxonomy. However, the evaluation of the questions was confined to measuring the qualitative satisfaction of domain experts and students. None of the question generators tested the questions on students and analysed the quality of the auto-generated questions by examining the question's difficulty, and the question's ability to discriminate between high ability and low ability students. The lack of quantitative analysis resulted in having no evidence on the quality of questions, and how the quality is affected by the ontology-based generation strategies, and the level of question in Bloom's taxonomy (determined by the question's stem templates). This paper presents an experiment carried out to address the drawbacks mentioned above by achieving two objectives. First, it assesses the auto-generated questions' difficulty, discrimination, and reliability using two statistical methods: Classical Test Theory (CTT) and Item Response Theory (IRT). Second, it studies the effect of the ontology-based generation strategies and the level of the questions in Bloom's taxonomy on the quality of the questions. This will provide guidance for developers and researchers working in the field of ontology-based question generators, and help building a prediction model using machine learning techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating the quality of the ontology-based auto-generated questions

Demaidi

Gaber

Filer

2017

Smart Learn. Environ.

View full text Add to dashboard Cite

show abstract

“…Note that the lower perplexity indicates the generated questions are more likely to human writing. Besides, we also use human evaluation that is frequently adopted in recent works about question generation [5,22,10]. In human evaluation, all generated questions mixed together are evaluated by human, and each question is rated by two people.…”

Section: Methodsmentioning

confidence: 99%

A novel framework for Automatic Chinese Question Generation based on multi-feature neural network model

et al. 2018

View full text Add to dashboard Cite

Automatic question generation from text or paragraph is a great challenging task which attracts broad attention in natural language processing. Because of the verbose texts and fragile ranking methods, the quality of top generated questions is poor. In this paper, we present a novel framework Automatic Chinese Question Generation (ACQG) to generate questions from text or paragraph. In ACQG, we use an adopted TextRank to extract key sentences and a template-based method to construct questions from key sentences. Then a multi-feature neural network model is built for ranking to obtain the top questions. The automatic evaluation result reveals that the proposed framework outperforms the state-of-the-art systems in terms of perplexity. In human evaluation, questions generated by ACQG rate a higher score.

show abstract

“…Unlike this previous research, the aim of the present study is to control item difficulty in the automatic question generation task. Automatic question generation has been proven to be able to generate questions with a quality that is comparable to those made by humans (Susanti et al 2017). Moreover, they cannot be distinguished from human-made questions (Le and Pinkwart 2015; Susanti et al 2017).…”

Section: Introductionmentioning

confidence: 99%

Controlling item difficulty for automatic vocabulary question generation

Susanti

Tokunaga

Nishikawa

et al. 2017

RPTEL

Self Cite

View full text Add to dashboard Cite

The present study investigates the best factor for controlling the item difficulty of multiple-choice English vocabulary questions generated by an automatic question generation system. Three factors are considered for controlling item difficulty: (1) reading passage difficulty, (2) semantic similarity between the correct answer and distractors, and (3) the distractor word difficulty level. An experiment was conducted by administering machine-generated items to three groups of English learners. The groups were determined based on their standardised English test scores. In total, 120 items, generated using combinations of the above three factors, were tested. The results reveal that the distractor word difficulty level had the greatest impact on item difficulty, but this tendency changed depending on the proficiency of the test takers. These results will be of use when implementing a fully automatic system for administrating tests.

show abstract

Evaluation of automatically generated English vocabulary questions

Cited by 15 publications

References 13 publications

Evaluating the quality of the ontology-based auto-generated questions

Evaluating the quality of the ontology-based auto-generated questions

A novel framework for Automatic Chinese Question Generation based on multi-feature neural network model

Controlling item difficulty for automatic vocabulary question generation

Contact Info

Product

Resources

About