Covariate Measurement Error Correction for Student Growth Percentiles Using the SIMEX Method

Shang, Yi; VanIwaarden, Adam; Betebenner, Damian W.

doi:10.1111/emip.12058

Cited by 15 publications

(32 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…74–75) agree with our choice and remark that our target “is more likely to be of interest” than the alternative. Willett () also uses our target and it is current convention in student growth research (e.g., Akram, Erickson, & Meyer, ; Lockwood & Castellano, ; McCaffrey et al., ; Monroe & Cai, ; Shang, Van Iwaarden, & Betebenner, ).…”

Section: Defining the Measures And Estimators Of Interestmentioning

confidence: 99%

Comparing the Accuracy of Student Growth Measures

Castellano

McCaffrey

2019

J Educational Measurement

View full text Add to dashboard Cite

Testing programs are often interested in using a student growth measure. This article presents analytic derivations of the accuracy of common student growth measures on both the raw scale of the test and the percentile rank scale in terms of the proportional reduction in mean squared error and the squared correlation between the estimator and target. The study contrasts the accuracy of the growth measures against that of current status measures—current test scores and their percentile ranks. Key findings include the extent that status measures are more accurate than any of the growth measures and that alternative methods to estimate growth could be more accurate than the currently used methods. Our findings highlight the importance for evaluating the statistical properties of growth measures along with other concerns for states that are debating the reporting of growth. Our results also point out that assessing the accuracy of growth measures requires the specification of quantities of interest in terms of latent achievement rather than observed test scores, which is common practice for developing status measures but essentially never done by testing programs for growth measures.

show abstract

Section: Defining the Measures And Estimators Of Interestmentioning

confidence: 99%

Comparing the Accuracy of Student Growth Measures

Castellano

McCaffrey

2019

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Presumably, practitioners interested in SGP are interested in the relative status of students' latent achievement as opposed to the relative status of students' observed scaled scores. Thus, many researchers (e.g., Akram et al., ; Lockwood & Castellano, ; Shang et al., ) have posited the “True SGP” as the target when estimating SGP, where the True SGP is the percentile rank of a student's current true score in the conditional distribution given the student's prior year true score. From our distributional assumptions, it follows that the conditional distribution of the current true scores given prior true scores is:

X_{2 i j} ∣ X_{1 i j} \sim N false(β X_{1 i j}, σ_{ε}^{2} = σ_{X_{2}}^{2} (1 - β^{2}) false)

, where

ε_{i j}

is the student‐level residual,

ε_{i j} = X_{2 i j} - β X_{1 i j}

.…”

Section: Research Question 1: Standard Mgpmentioning

confidence: 99%

“…Shang et al. () investigated such an ME‐adjustment approach for SGP using “simulation‐extrapolation” (SIMEX; Carroll, Ruppert, Stefanski, & Crainiceanu, ; Cook & Stefanski, ). They use the SIMEX method to obtain an estimate of the conditional distribution of the current observed test score that is corrected for ME, and then use this distribution to rank the student's current score to obtain an SGP.…”

Section: Research Question 2: Alternativesmentioning

confidence: 99%

“…Under our assumptions of normally distributed data and large samples, the Shang et al. () approach yields an estimate of the conditional distribution of observed current score W 2 given true prior score X 1 evaluated at observed scores w 2 and w 1 :

{SGP}_{SIMEX} = F_{W_{2} ∣ X_{1}} false(W_{2} = w_{2} ∣ X_{1} = w_{1} false) = Φ (\frac{W_{2} - β W_{1}}{\sqrt{σ_{ε}^{2} + σ_{U_{2}}^{2}}})

, or the percentile rank of W 2 in a normal distribution that has mean

β W_{1}

and variance

σ_{ε}^{2} + σ_{U_{2}}^{2}

. See the online Supporting Information, Appendix B.1.1, for additional details on the SIMEX method.…”

Section: Research Question 2: Alternativesmentioning

confidence: 99%

See 1 more Smart Citation

The Accuracy of Aggregate Student Growth Percentiles as Indicators of Educator Performance

Castellano

McCaffrey

2017

Educational Measurement

View full text Add to dashboard Cite

Mean or median student growth percentiles (MGPs) are a popular measure of educator performance, but they lack rigorous evaluation. This study investigates the error in MGP due to test score measurement error (ME). Using analytic derivations, we find that errors in the commonly used MGP are correlated with average prior latent achievement: Teachers with low prior achieving students have MGPs that underestimate true teacher performance and vice versa for teachers with high achieving students. We evaluate alternative MGP estimators, showing that aggregates of SGP that correct for ME only contain errors independent of prior achievement. The alternatives are thus more fair because they are not biased by prior mean achievement and have smaller overall variance and larger marginal reliability than the Standard MGP approach. In addition, the mean estimators always outperform their median counterparts.

show abstract

“…The original methodological framework for SGPs is quantile regression (QR), and an R package (Betebenner, VanIwaarden, Domingue, & Shang, ) has been developed in support of the methodology. Within this framework, which has been the focus of several recent research efforts (e.g., Castellano & Ho, ; McCaffrey, Castellano, & Lockwood, ; Shang, VanIwaarden, & Betebenner, ), SGPs are calculated in multiple steps. First, student scores are generated for each year's test.…”

mentioning

confidence: 99%

Examining the Reliability of Student Growth Percentiles Using Multidimensional IRT

Monroe¹,

Cai

2015

Educational Measurement

View full text Add to dashboard Cite

Student growth percentiles (SGPs, Betebenner, 2009) are used to locate a student's current score in a conditional distribution based on the student's past scores. Currently, following Betebenner (2009), quantile regression (QR) is most often used operationally to estimate the SGPs. Alternatively, multidimensional item response theory (MIRT) may also be used to estimate SGPs, as proposed by Lockwood and Castellano (2015). A benefit of using MIRT to estimate SGPs is that techniques and methods already developed for MIRT may readily be applied to the specific context of SGP estimation and inference. This research adopts a MIRT framework to explore the reliability of SGPs. More specifically, we propose a straightforward method for estimating SGP reliability. In addition, we use this measure to study how SGP reliability is affected by two key factors: the correlation between prior and current latent achievement scores, and the number of prior years included in the SGP analysis. These issues are primarily explored via simulated data. In addition, the QR and MIRT approaches are compared in an empirical application.

show abstract

Covariate Measurement Error Correction for Student Growth Percentiles Using the SIMEX Method

Cited by 15 publications

References 27 publications

Comparing the Accuracy of Student Growth Measures

Comparing the Accuracy of Student Growth Measures

The Accuracy of Aggregate Student Growth Percentiles as Indicators of Educator Performance

Examining the Reliability of Student Growth Percentiles Using Multidimensional IRT

Contact Info

Product

Resources

About