Using growth models to monitor school performance: comparing the effect of the metric and the assessment

Abstract

This paper investigates whether inferences about school performance based on longitudinal models are consistent when different assessments and metrics are used as the basis for analysis. Using norm-referenced (NRT) and standards-based (SBT) assessment results from panel data of a large heterogeneous school district, we examine inferences based on vertically equated scale scores, normal curve equivalents (NCEs), and nonvertically equated scale scores. The results indicate that the effect of the metric depends upon the evaluation objective. NCEs significantly underestimate absolute individual growth, but NCEs and scale scores yield highly correlated (r 4.90) school-level results based on mean initial status and growth estimates. SBT and NRT results are highly correlated for status but only moderately correlated for growth. We also find that as few as 30 students per school provide consistent results and that mobility tends to affect inferences based on status but not growth -irrespective of the assessment or metric used.

The association of Latino children’s kindergarten school readiness profiles with Grade 2–5 literacy achievement trajectories.

“…An unintended consequence of using NCEs in growth modeling is that it could result in flat trajectories for students whose ranking among their peers remained the same across time, despite the fact that their knowledge base grew. That is, flat growth trajectories with NCEs reflect normative growth (for more on this see Goldschmidt, Choi, Martinez, & Novak, 2010). Internal consistency coefficients for the E-LA subtest ranged from .92 to .94 for Latino students in Grades 2–5 (California Department of Education Standards & Assessment Division, 2010–2013).…”

Change in home language environment and English literacy achievement over time: A multi-group latent growth curve modeling investigation

“…As for the last point, the z-score metric captured students’ relative growth compared to peers at the time of testing but is not truly sufficient for measuring growth as, to measure each student’s growth independent of their peer-to-peer comparison, the underlying assessments across the years must be vertically scaled. Psychometrically, the metric scale utilized in LGCM analyses can greatly influence the results associated with growth trajectories (Goldschmidt et al., 2010). Modeling individual students’ growth trajectories in academic achievement requires vertically scaled test scores for consistent interpretation across time points (Briggs et al, 2008).…”

Analysis of Accountability Ratings for Elementary Schools Texas, USA: Considering the School Growth and Students’ Demographics

“…The lack of randomization of the current accountability systems has serious limitations with regard to the available data about instructional practices, teacher characteristics, and students' characteristics (Linn, 2006). Education policymakers are beginning to realize that inferences based on school accountability systems that use single-year comparisons of school performance aggregate data simply reflect accumulated inequities and enrolment characteristics, and not necessarily the school effectiveness in influencing student academic progress (Goldschmidt, Choi, Martinez, & Novak, 2010). A status oriented accountability approach is well intentioned, but it may be unfair as a method of determining the effectiveness of a district or school because populations with different demographics are not equal (Marzano & Waters, 2009).…”