A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Abstract

Value-added models (VAMs) are used by many states to assess contributions of individual teachers and schools to students' academic growth. The generalized persistence VAM, one of the most flexible in the literature, estimates the "value added" by individual teachers to their students' current and future test scores by employing a mixed model with a longitudinal database of test scores. There is concern, however, that missing values that are common in the longitudinal student scores can bias value-added assessments, especially when the models serve as a basis for personnel decisions -such as promoting or dismissing teachers -as they are being used in some states. Certain types of missing data require that the VAM be modeled jointly with the missingness process in order to obtain unbiased parameter estimates.This dissertation studies two problems. First, the flexibility and multimembership random effects structure of the generalized persistence model lead to computational challenges that have limited the model's availability. To this point, no methods have been developed for scalable maximum likelihood estimation of the model. An EM algorithm to compute maximum likelihood estimates efficiently is developed, making use of the sparse structure of the random effects and error covariance matrices. The algorithm is implemented in the package GPvam in R statistical software. Illustrations of the gains in computational efficiency achieved by the estimation procedure are given.Furthermore, to address the presence of potentially nonignorable missing data, a flexible correlated random effects model is developed that extends the generalized persistence model to jointly model the test scores and the missingness process, allowing the process to depend on both students and teachers. The joint model gives the ability to test the sensitivity of the VAM to the presence of noni ignorable missing data. Estimation of the model is challenging due to the nonhierarchical dependence structure and the resulting intractable high-dimensional integrals. Maximum likelihood estimation of the model is performed using an EM algorithm with fully exponential Laplace approximations for the E step. The methods are illustrated with data from university calculus classes and with data from standardized test scores from an urban school district.ii To Laura, Oliver, and the rest of my family.iii ACKNOWLEDGEMENTS I owe a heartfelt thanks to my advisors, Dr. Lohr and Dr. Yang, who have dedicated so much of their time and effort to helping me develop this dissertation.They were selfless in their willingness to respond quickly to all of my emails and to meet with me whenever I needed their assistance. I am indebted to them for significant portions of my graduate education that they went out of their way to provide me with, including encouragement and funding to travel to JSM in Vancouver and Miami Beach, help with job searching, guiding me through the publication and peer-review process, and in general teaching me how to work in an academic environme...

Value-Added Modeling and Educational Accountability

“…However, as found by Herrmann et al (2013), results were inconclusive as to whether fixed or random approaches were preferable when the question of interest was the percentage of teachers who would shift from lowest to highest quintile, or vice versa, over 2 years. Karl, Yang, and Lohr (2013) found a correlated random effects model can be useful for investigating the impact of missingness on VA teacher rankings. In another study, Guarino, Reckase, and Wooldridge (2012) found that no estimation approach is better in all situations.…”

The Implications of Reduced Testing for Teacher Accountability

“…That is, teacher effectiveness estimates in which data are missing at random compared to those in which data are not missing at random correlate at or above 0.98. In another study, Karl, Yang, and Lohr (2013) used flexible correlated random-effects models to jointly model student responses and missing data indicators. Their findings suggest that teacher rankings are sensitive to missing data under some models.…”

The Sensitivity of College Football Rankings to Several Modeling Choices

“…Rizopoulos et al (2009) use this method to estimate the parameters of a joint model for a continuous longitudinal process and a time-todropout measurement. Karl (2012a) applies this approach to a multi-membership joint model. We will give a brief overview of the EM estimation procedure, and refer to Karl (2012a) for further details, as well as to Rizopoulos et al (2009) for similar calculations made in the setting of nested random effects.…”

“…We combine the FBS and FCS files, remove all games labeled "away", remove games between FCS and lower division teams, purge redundant neutral site games, add an indicator for games played between FBS and FCS schools, and remove the records of games played after the production of the final BCS rankings in each year. The processed data are available from Karl (2012b). See Table 12 in Appendix B for the first observations of our 2008 data set.…”