A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats

Abstract

A meta-analysis was conducted on the effects of multiple-choice and open-ended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.

Assessing writing ability in primary education : on the evaluation of text quality and text complexity

“…As Table 13 shows, no significant differences were found. Similar to earlier findings by Hohensinn and Kubinger (2011) and In'nami and Koizumi (2009), the CR items seemed slightly more difficult.…”

“…Lissitz and Hou (2012) argued that different response formats do not necessarily elicit the same expression of skills and that CR items may assess skills in a different manner than MC items do. Lastly, MC items were generally found easier then CR items (e.g., In'nami & Koizumi, 2009). In the present study, the validity of both multiple-choice and constructed response revision items is evaluated, as well as the differences in test characteristics of both formats.…”

Progression paths in children’s problem solving: The influence of dynamic testing, initial variability, and working memory

“…Furthermore, our analogy tasks were openended; children had to construct their answers so enabling us to undertake fine-grained analyses of children's solving processes (e.g., Harpaz-Itay, Kaniel, & Ben-Amram, 2006;Resing & Elliott, 2011;Tzuriel & Galinka, 2000). Although we know that such tasks are more difficult than multiple choice tasks (e.g., Behuniak, Rogers, & Dirir, 1996;In'nami & Kozumi, 2009;Martinez, 1999), and individuals require more help during training to solve these tasks (Stevenson et al, 2016), these type of tasks may improve learning after extensive instruction.…”

Brain Dominance and Test Format: A Case of Vocabulary

“…Indubitably, test format affects test performance (Shohamy, 1984;Wolf, 1993;Kobayashi, 2002;Bowles & Salthouse, 2008;In'nami & Koizumi, 2009). Bachman and Palmer (1996, p. 46) are of the opinion that "the characteristics of the tasks used are always likely to affect test scores to some degree, so that there is virtually no test that yields only information about the ability we want to measure".…”