Mean Effects of Test Accommodations for ELLs and Non-ELLs: A Meta-Analysis of Experimental Studies

Abstract

The objective was to examine the impact of different types of accommodations on performance in content tests such as mathematics. The meta‐analysis included 14 U.S. studies that randomly assigned school‐aged English language learners (ELLs) to test accommodation versus control conditions or used repeated measures in counter‐balanced order. Individual effect sizes (Glass's d) were calculated for 50 groups of ELLs and 32 groups of non‐ELLs. Individual effect sizes for English language and native language accommodations were classified into groups according to type of accommodation and timing conditions. Means and standard errors were calculated for each category. The findings suggest that accommodations that require extra printed materials need generous time limits for both the accommodated and unaccommodated groups to ensure that they are effective, equivalent in scale to the original test, and therefore more valid owing to reduced construct‐irrelevant variance. Computer‐administered glossaries were effective even when time limits were restricted. Although the Plain English accommodation had very small average effect sizes, inspection of individual effect sizes suggests that it may be much more effective for ELLs at intermediate levels of English language proficiency. For Spanish‐speaking students with low proficiency in English, the Spanish test version had the highest individual effect size (+1.45).

On the Effectiveness of Pop-Up English Language Glossary Accommodations for EL Students in Large-Scale Assessments

“…Meta-analyses have found large effect sizes for pop-up glossaries, indicating their promise as an easy-to-use, effective, and valid accommodation for ELs to demonstrate their content knowledge without the constraint of linguistic difficulty (e.g., Abedi, Hofstetter, & Lord, 2004;Pennock-Roman & Rivera, 2011). However, it remains difficult to generalize these results.…”

“…Because the test used was already extant, this removed the need to assemble a test and thus reduced the risk of administering an assessment that contained material students either hadn't been exposed to or was too difficult for them, as observed by Kiplinger et al (2000). Pennock-Roman and Rivera (2011) noted that small effect sizes for given accommodations may not only reflect that accommodation's effectiveness, but its effectiveness relative to the difficulty of the test. We note that our study included a variety of item difficulties, including some that most ELs got correct.…”

“…Pennock-Roman and Rivera (2011) revealed that dictionaries had a higher mean effect size for ELs when they were used with less restrictive time limits, and that effect sizes were more varied with more restrictive time limits. Furthermore, multiple studies have failed to find a significant effect for customized dictionaries.…”

“…Abedi, Courtney, and Leon (2003a) found that while grade 4 ELs saw no significant score changes for the linguistic modification accommodation, grade 8 ELs had a significant increase in their test scores. Furthermore, a meta-analysis by Pennock-Roman and Rivera (2011) found that when studies took English language proficiency (ELP) into account, linguistic modification had a larger mean effect size for ELs classified as high and high intermediate, which includes students who are either fluent or near fluent in the English language, respectively. Further research could delve into exactly which aspects of a test's linguistic structure need to be modified on a test to improve the accommodation's effectiveness and while not influencing the test's construct-related validity.…”

Toward Accessible Assessments: The Promises and Limitations of Test Item Adaptations for Students with Disabilities and English Language Learners

“…In addition, a number of states have implemented "simplified language" versions of general test forms at some points during NCLB (e.g., see Kopriva, 2008b;Pennock-Roman & Rivera, 2011). These alternative test forms were meant to measure the same measurement target as those on one or more general test forms, and were seen as an accommodation for some ELL or SwDs who were at grade level in their schooling but did not have the language capacity to meaningfully engage with the language on the test forms administered to the majority of students (DePascale, 2010a;Kieffer, Lesaux, Rivera, & Francis, 2009;Sireci, Li, & Scarpati, 2003).…”