Evaluating the Effectiveness of a Full‐population Estimation Method

Abstract

At present, although the percentages of students with disabilities (SDs) and/or students who are English language learners (ELL) excluded from a NAEP administration are reported, no statistical adjustment is made for these excluded students in the calculation of NAEP results. However, the exclusion rates for both SD and ELL students vary substantially across jurisdictions at a given administration, and, in some cases, have changed substantially over time within a jurisdiction. Consequently, comparisons of performance based on reported NAEP scores may indeed be biased by differential exclusion and identification practices. Using only NAEP data, this report investigates plausible explanations for the observed heterogeneity among jurisdictions in exclusion rates. It also examines the operating characteristics of a particular class of methods that carry out statistical adjustments to NAEP's reported scores to address the possible bias due to differential exclusion rates. The final results of such adjustments are termed full‐population estimates (FPEs). The conclusions are that there is both a strong likelihood of bias and that neither the current NAEP procedure nor the FPE methodologies constitutes an ideal solution. The former because it assumes that all excluded students could not meaningfully participate in NAEP, and the latter because they implicitly assume that all students could obtain a proper NAEP score.

Large-Scale Group-Score Assessment

“…McLaughlin (2000McLaughlin ( , 2005 proposed a regression approach by imputing excluded students' proficiencies from other available data. McLaughlin's work was further developed by Braun et al (2008).…”

An Investigation of Bias in Reports of the National Assessment of Educational Progress

“…Some of the variation is due to small sample fluctuations. To generate more stable estimates, we employed an empirical Bayes-type approach to derive smoothed category-specific exclusion rates (for details, see Braun, Zhang, & Vezzu, 2008).…”

“…For the SD-all case, we deleted three variables. For the ELL-only case, we deleted two variables (for further details, consult Braun et al, 2008).…”

“…For further information on the methodology and more extensive numerical results, the interested reader should consultBraun, Zhang, and Vezzu (2008).…”

chapter 13 Large‐Scale Indicator Assessments: What Every Educational Policymaker Should Know

“…Nevertheless, an increase in inclusion rates of special needs students remains a major hurdle for indicator assessments. In exploring alternative solutions so as to avoid biased scores caused by exclusion, researchers developed statistical methods (such as “full population estimates”) that adjust the scores to mimic the estimates that would have been obtained had everyone in the sample actually been assessed (Braun, Zhang, & Vezzu, 2006, p. 2; McLaughlin, 2005). While such approaches are not perfect, the full populations estimate method is a viable option for improving the generalizability of indicator systems (Wise, 2003).…”