Conducting Simulation Studies in Psychometrics

Feinberg, Richard A.; Rubright, Jonathan D.

doi:10.1111/emip.12111

Cited by 64 publications

(50 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In terms of topical areas of study, most of the published meta‐analyses came from three primary subject areas: test format effects, test accommodations, and the predictive validity of operational testing programs. Though the use of simulation techniques in educational measurement applications has become an important and popular tool (Feinberg & Rubright, ), we found no meta‐analyses of such techniques in our sample. We speculate that the lack of utilization of the meta‐analytic methodology may largely be due to an unawareness of this approach to conducting research, and/or to a dearth of primary studies on similar topics necessary to conduct an adequately powered meta‐analysis.…”

Section: Discussionmentioning

confidence: 60%

A Topical and Methodological Systematic Review of Meta‐Analyses Published in the Educational Measurement Literature

Rios

Ihlenfeldt

Dosedel

et al. 2019

Educational Measurement

View full text Add to dashboard Cite

This systematic review investigated the topics studied and reporting practices of published meta‐analyses in educational measurement. Our findings indicated that meta‐analysis is not a highly utilized methodological tool in educational measurement; on average, less than one meta‐analysis has been published per year over the past 30 years (28 meta‐analyses were published between 1986 and 2016). Within the field, researchers have utilized meta‐analysis to study three primary subject areas: test format effects, test accommodations, and predictive validity of operational testing programs. In regard to reporting practices, authors often failed to provide descriptive details of both their search strategy and sample characteristics limiting reproducibility and generalizability of findings, respectively. Furthermore, diagnostic analyses of outliers, publication bias, and statistical power were not provided for the majority of studies, putting into question the validity of inferences made from the meta‐analyses sampled. The lack of transparent and replicable practices of meta‐analyses in educational measurement is a concern for generating credible research syntheses that can assist the field in improving evidence‐based practices. Recommendations are provided for improving training and editorial standards of meta‐analytic research.

show abstract

Section: Discussionmentioning

confidence: 60%

A Topical and Methodological Systematic Review of Meta‐Analyses Published in the Educational Measurement Literature

Rios

Ihlenfeldt

Dosedel

et al. 2019

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…In terms of topical areas of study, most of the published meta-analyses came from three primary subject areas: test format effects, test accommodations, and the predictive validity of operational testing programs. Though the use of simulation techniques in educational measurement applications has become an important and popular tool (Feinberg & Rubright, 2016), we found no meta-analyses of such techniques in our sample. We speculate that the lack of utilization of the meta-analytic methodology may largely be due to an unawareness of this approach to conducting research, and/or to a dearth of primary studies on similar topics necessary to conduct an adequately powered meta-analysis.…”

Section: Discussionmentioning

confidence: 69%

A Topical and Methodological Systematic Review of Meta-Analyses Published in the Educational Measurement Literature

Rios¹,

Ihlenfeldt²,

Dosedel³

et al. 2019

Preprint

View full text Add to dashboard Cite

This systematic review investigated the topics studied and reporting practices of published meta-analyses in educational measurement. Our findings indicated that meta-analysis is not a highly utilized methodological tool in educational measurement; on average, less than one meta-analysis has been published per year over the past 30 years (28 meta-analyses were published between 1986 and 2016). Within the field, researchers have utilized meta-analysis to study three primary subject areas: test format effects, test accommodations, and predictive validity of operational testing programs. In regard to reporting practices, authors often failed to provide descriptive details of both their search strategy and sample characteristics limiting reproducibility and generalizability of findings, respectively. Furthermore, diagnostic analyses of outliers, publication bias, and statistical power were not provided for the majority of studies, putting into question the validity of inferences made from the meta-analyses sampled. The lack of transparent and replicable practices of meta-analyses in educational measurement is a concern for generating credible research syntheses that can assist the field in improving evidence-based practices. Recommendations are provided for improving training and editorial standards of meta-analytic research.

show abstract

“…Several measures can be used in simulation studies to identify whether a statistic tends to be accurate (Feinberg and Rubright, 2016). Here, there are two sets of continuous measures: the true growth values used to create the data and the estimated values.…”

Section: Resultsmentioning

confidence: 99%

Some Limits Using Random Slope Models to Measure Academic Growth

Wright

2017

Front. Educ.

View full text Add to dashboard Cite

Academic growth is often estimated using a random slope multilevel model with several years of data. However, if there are few time points, the estimates can be unreliable. While using random slope multilevel models can lower the variance of the estimates, these procedures can produce more highly erroneous estimates-zero and negative correlations with the true underlying growth-than using ordinary least squares estimates calculated for each student or school individually. An example is provided where schools with increasing graduation rates are estimated to have negative growth and vice versa. The estimation is worse when the underlying data are skewed. It is recommended that there are at least six time points for estimating growth if using a random slope model. A combination of methods can be used to avoid some of the aberrant results if it is not possible to have six or more time points.

show abstract

Conducting Simulation Studies in Psychometrics

Cited by 64 publications

References 41 publications

A Topical and Methodological Systematic Review of Meta‐Analyses Published in the Educational Measurement Literature

A Topical and Methodological Systematic Review of Meta‐Analyses Published in the Educational Measurement Literature

A Topical and Methodological Systematic Review of Meta-Analyses Published in the Educational Measurement Literature

Some Limits Using Random Slope Models to Measure Academic Growth

Contact Info

Product

Resources

About