Multidimensional Assessment of Value Added by Teachers to Real-World                    Outcomes

Broatch, Jennifer; Lohr, Sharon L.

doi:10.3102/1076998610396900

Cited by 21 publications

(25 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In particular, studies should address issues related to establishing a "good" student baseline, identifying minimum data requirements, and selecting appropriate measures of student success. Additionally, the proposed methods should be extended for use with a multidimensional VAM (Broatch and Lohr 2012) to simultaneously estimate the effects of a program on student achievement test scores and "real-world" outcomes, such as college entry. By using the multidimensional model in this context, professional development programs would have methodology to more broadly define their effects on student success.…”

Section: Discussionmentioning

confidence: 99%

Defining Program Effects: A Distribution-Based Perspective

Green

Stroup

Fellers

2017

Statistics and Public Policy

View full text Add to dashboard Cite

In an age of accountability, it is critical to define and estimate the effects of teacher education and professional development programs on student learning in ways that allow stakeholders to explore potential reasons for what is observed and to enhance program quality and fidelity. Across the suite of statistical models used for program evaluation, researchers consistently measure program effectiveness using the coefficients of fixed program effects. We propose that program effects are best characterized not as a single effect to be estimated, but as a distribution of teacher-specific effects. In this article, we first discuss this approach and then describe one way it could be used to define and estimate program effects within a value-added modeling context. Using an example dataset, we demonstrate how program effect estimates can be obtained using the proposed methodology and explain how distributions of these estimates provide additional information and insights about programs that are not apparent when only looking at average effects. By examining distributions of teacher-specific effects as proposed, researchers have the opportunity to more deeply investigate and understand the effects of programs on student success.

show abstract

Section: Discussionmentioning

confidence: 99%

Defining Program Effects: A Distribution-Based Perspective

Green

Stroup

Fellers

2017

Statistics and Public Policy

View full text Add to dashboard Cite

show abstract

“…Thus, RealVAMS uses sparse matrix routines provided by Matrix (Bates and Maechler, 2015). Broatch and Lohr (2012) use the pseudo-likelihood approach to obtain approximations to the maximum likelihood estimators; they then adopt the penalized quasi-likelihood approach in SAS PROC GLIMMIX (SAS Institute Inc., 2008) to approximate maximum likelihood estimates (Breslow and Clayton, 1993;Wolfinger and O'Connell, 1993). However, the SAS estimation in presented in Broatch and Lohr (2012) is limited to less than 30 random effects, which is not feasible or efficient for a large scale model.…”

Section: Normal Multimembership Modelmentioning

confidence: 99%

“…Originally developed in Broatch and Lohr (2012), the RealVAMS model is a multivariate generalized linear mixed model (GLMM) that extends previous theory by allowing the simultaneous joint modeling of continuous and binary outcomes within a value-added framework. Although the model may be used in a variety of contexts (e.g., sports, medicine, etc.…”

Section: Introductionmentioning

confidence: 99%

RealVAMS: An R Package for Fitting a Multivariate Value-added Model (VAM)

Broatch¹,

Green²,

Karl³

2018

The R Journal

Self Cite

View full text Add to dashboard Cite

We present RealVAMS, an R package for fitting a generalized linear mixed model to multimembership data with partially crossed and partially nested random effects. RealVAMS utilizes a multivariate generalized linear mixed model with pseudo-likelihood approximation for fitting normally distributed continuous response(s) jointly with a binary outcome. In an educational context, the model is referred to as a multidimensional value-added model, which extends previous theory to estimate the relationships between potential teacher contributions toward different student outcomes and to allow the consideration of a binary, real-world outcome such as graduation. The simultaneous joint modeling of continuous and binary outcomes was not available prior to RealVAMS due to computational difficulties. In this paper, we discuss the multidimensional model, describe RealVAMS, and demonstrate the use of this package and its modeling options with an educational data set.

show abstract

“…At worst, the (fully exponential) Laplace approximation presents an improvement over PQL in this setting, even if consistency is not guaranteed. PQL is widely used -such as in SAS PROC GLIMMIX -despite the the fact that it produces potentially biased estimates, because PQL makes some problems computationally feasible that otherwise would not be (Broatch and Lohr, 2011). Even the bias-corrected PQL of Lin and Breslow (1996) is inconsistent (Jiang, 2007).…”

Section: Approximation Errormentioning

confidence: 99%

A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Karl¹,

Yang

Lohr

2013

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

Value-added models (VAMs) are used by many states to assess contributions of individual teachers and schools to students' academic growth. The generalized persistence VAM, one of the most flexible in the literature, estimates the "value added" by individual teachers to their students' current and future test scores by employing a mixed model with a longitudinal database of test scores. There is concern, however, that missing values that are common in the longitudinal student scores can bias value-added assessments, especially when the models serve as a basis for personnel decisions -such as promoting or dismissing teachers -as they are being used in some states. Certain types of missing data require that the VAM be modeled jointly with the missingness process in order to obtain unbiased parameter estimates.This dissertation studies two problems. First, the flexibility and multimembership random effects structure of the generalized persistence model lead to computational challenges that have limited the model's availability. To this point, no methods have been developed for scalable maximum likelihood estimation of the model. An EM algorithm to compute maximum likelihood estimates efficiently is developed, making use of the sparse structure of the random effects and error covariance matrices. The algorithm is implemented in the package GPvam in R statistical software. Illustrations of the gains in computational efficiency achieved by the estimation procedure are given.Furthermore, to address the presence of potentially nonignorable missing data, a flexible correlated random effects model is developed that extends the generalized persistence model to jointly model the test scores and the missingness process, allowing the process to depend on both students and teachers. The joint model gives the ability to test the sensitivity of the VAM to the presence of noni ignorable missing data. Estimation of the model is challenging due to the nonhierarchical dependence structure and the resulting intractable high-dimensional integrals. Maximum likelihood estimation of the model is performed using an EM algorithm with fully exponential Laplace approximations for the E step. The methods are illustrated with data from university calculus classes and with data from standardized test scores from an urban school district.ii To Laura, Oliver, and the rest of my family.iii ACKNOWLEDGEMENTS I owe a heartfelt thanks to my advisors, Dr. Lohr and Dr. Yang, who have dedicated so much of their time and effort to helping me develop this dissertation.They were selfless in their willingness to respond quickly to all of my emails and to meet with me whenever I needed their assistance. I am indebted to them for significant portions of my graduate education that they went out of their way to provide me with, including encouragement and funding to travel to JSM in Vancouver and Miami Beach, help with job searching, guiding me through the publication and peer-review process, and in general teaching me how to work in an academic environme...

show abstract

Multidimensional Assessment of Value Added by Teachers to Real-World Outcomes

Cited by 21 publications

References 32 publications

Defining Program Effects: A Distribution-Based Perspective

Defining Program Effects: A Distribution-Based Perspective

RealVAMS: An R Package for Fitting a Multivariate Value-added Model (VAM)

A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Contact Info

Product

Resources

About