Understanding and evaluating the SAS® EVAAS® Univariate Response Model (URM) for measuring teacher effectiveness

Methodological Concerns About the Education Value-Added Assessment System (EVAAS): Validity, Reliability, and Bias

“…The better the test data available, even if taken from different types of standardized achievement tests (e.g., large-scale standardized tests that are aligned to either national or specific state standards, again, as per NCLB ( 2001)), the better the model used (i.e., the URM), and the better (i.e., more valid, reliable, and unbiased) the model estimates (W. L. Sanders, 2003Sanders, , 2006Wright et al, 2006Wright et al, , 2010. However, it should be noted here that recent evidence offered in Vosters et al (2018) suggests that the EVAAS's URM may be inferior to the EVAAS's aforementioned MRM, if one is to deduce from Vosters et al (2018) that the MRM is akin to the other VAMs to which Vosters et al (2018) Regardless, growth scores are ultimately aggregated to yield teacher-level value-added estimates. Depending on where teachers' EVAAS estimates fall, as compared with the similar teachers to whom they are compared (e.g., within districts), EVAAS modelers categorize teachers hierarchically along a continuum (W. L. Sanders, 2003Sanders, , 2006W.…”

“…Although scholars have conducted similar research on VAMs and the EVAAS, no external researchers of whom we are aware have published studies based on actual EVAAS output. Rather, researchers have used simulation data or other data to replicate EVAAS estimates using that which is publicly available about model specifics (see, for example, McCaffrey et al, 2004;Vosters et al, 2018; see also W. L. Sanders et al, 2009; W. L. Sanders & Wright, 2008).…”

“…While authors of these simulation studies have, according to EVAAS developers, evidenced that the EVAAS model is indeed "robust" (W. L. Sanders & Wright, 2008, p. 1), and authors of these studies have made strides in terms of the further development of the EVAAS as well as other VAMs in general, authors of these same studies have also raised legitimate concerns about the EVAAS. Subsequently, having been debated for over one decade now are issues including but not limited to those pertaining to the actual robustness and, hence, accuracy of EVAAS output, primarily surrounding: (a) missing data and the extent to which data not missing at random might cause selection bias (see also ; (b) bias, as related but caused by the lack of statistical controls used, or not typically used with the EVAAS (see, for example, Amrein-Beardsley, 2014; W. L. Sanders & Wright, 2008), that even if used may not be sophisticated enough to effectively counter the biasing effects of extraneous, student demographic variables that contaminate teachers valueadded estimates (see Paufler & Amrein-Beardsley, 2014, in press; see also Koedel et al, 2015) about whether unbiased estimates might ever be produced by either EVAAS model or any such VAM, and especially the EVAAS's univariate response model (URM; see also Chetty et al, 2014a;Rothstein, 2009Rothstein, , 2010Rothstein, , 2014Vosters et al, 2018); (c) the correct matching of teachers to their students, also proportionally when students are taught similar subjects by different teachers, taught in team teaching scenarios, and so on; (d) regression to the mean, as caused (or countered) by shrinkage estimation via best linear unbiased prediction (BLUP) or empirical Bayes estimation methods (see also McCaffrey et al, 2008;Raudenbush & Bryk, 2002); and the like. For a more comprehensive review of these issues, still of debate, please see Amrein-Beardsley (2008) and W. L. Sanders and Wright (2008).…”

Will Mentoring a Student Teacher Harm My Evaluation Scores? Effects of Serving as a Cooperating Teacher on Evaluation Metrics

“…Our sample includes 458,717 teacher-by-year observations. Table 1 presents descriptive information about types of evaluation data we have for teachers, including observation ratings and their value-added to student achievement measures (Teacher Value-Added Assessment System, or TVAAS; see Vosters, Guranio, and Wooldridge, 2018, for more information on how these scores are calculated). Similar to other states' use of value-added measures, TVAAS is calculated using state test data and intends to capture an individual teacher's effect on student achievement; teachers receive scores for specific tested subjects as well as composite scores.…”

“…This is because we do not have access to evaluation data for the period preceding serving as a cooperating teacher. Second, TVAAS models differ from traditional value-added models insofar that scores for each teacher are calculated separately for each student cohort and that teacher value-added are calculated using empirical Bayes’s estimates (Vosters et al, 2018).…”