Measuring non-native speakers' proficiency of English by using a test with automatically-generated fill-in-the-blank questions

Abstract

This paper proposes the automatic generation of Fill-in-the-Blank Questions (FBQs) together with testing based on Item Response Theory (IRT) to measure English proficiency. First, the proposal generates an FBQ from a given sentence in English. The position of a blank in the sentence is determined, and the word at that position is considered as the correct choice. The candidates for incorrect choices for the blank are hypothesized through a thesaurus. Then, each of the candidates is verified by using the Web. Finally, the blanked sentence, the correct choice and the incorrect choices surviving the verification are together laid out to form the FBQ. Second, the proficiency of nonnative speakers who took the test consisting of such FBQs is estimated through IRT.Our experimental results suggest that: (1) the generated questions plus IRT estimate the non-native speakers' English proficiency; (2) while on the other hand, the test can be completed almost perfectly by English native speakers; and (3) the number of questions can be reduced by using item information in IRT.The proposed method provides teachers and testers with a tool that reduces time and expenditure for testing English proficiency.

Detecting Context Dependence in Exercise Item Candidates Selected from Corpora

“…Previous work on exercise item generation has adopted different strategies for carrier sentence selection. In some cases, sentences are mainly required to contain a lexical item or a linguistic pattern that constitutes the target of the exercise, but context dependence is not explicitly addressed (Sumita et al, 2005;Arregik, 2011). Another alternative has been using dictionary examples as carrier sentences, e.g.…”

“…Such sentences are also known as seed sentences (Sumita et al, 2005) or carrier sentences (Smith et al, 2010) in the Intelligent ComputerAssisted Language Learning (ICALL) literature. Interest for the use of corpora in language learning has arisen already in the 1980s, since the increasing amount of digital text available enables learning through authentic language use (O'Keeffe et al, 2007).…”

RevUP: Automatic Gap-Fill Question Generation from Educational Texts

“…) with little attention given to the semantics and educational relevance of the questions (Heilman and Smith [2009], Mitkov et al [2006], Mostow and Chen [2009], Wolfe et al [1975], Wyse and Piwek [2009]). On the other hand, previous works in GFQG have generally worked with vocabularytesting and language learning (Smith et al [2010], Sumita et al [2005]). Smith et al presented TedClogg which took gap-phrases as input and found multiple choice distractors from a distributional thesaurus.…”

Automatic Generation of Challenging Distractors Using Context-Sensitive Inference Rules

“…Therefore it cannot be used with carrier sentences that do not contain strong collocations, such as the sentence in Figure 1. Sumita et al (2005) apply a simple web search approach to judge the reliability of an item. They check whether the carrier sentence with the target word replaced by the distractor can be found on the web.…”