Studying Score Stability with a Harmonic Regression Family: A Comparison of Three Approaches to Adjustment of Examinee‐Specific Demographic Data

In recent years, harmonic regression models have been applied to implement quality control for educational assessment data consisting of multiple administrations and displaying seasonality. As with other types of regression models, it is imperative that model adequacy checking and model fit be appropriately conducted. However, there has been no literature on how to perform a comprehensive model adequacy evaluation when applying harmonic regression models to sequential data with seasonality in the educational assessment field. This paper is intended to fill this gap with an illustration of real data from an English language assessment. Two types of cross-validation, leave-one-out and out-of-sample, were designed to measure prediction errors and check model validation. Three types of R-squared (R 2 , R 2 adj , and R 2 pred ) and various residual diagnostics were applied to check model adequacy and model fitting.

Section: Discussionmentioning

confidence: 99%

Model Adequacy Checking for Applying Harmonic Regression to Assessment Quality Control

Qian

2021

“…One was that the test center information was available for each test taker and was accurate. The other consideration was that prior studies (Lee & Haberman, , ) had found regional effects on test taker performance.…”

Section: Methodsmentioning

confidence: 99%

“…One was that the test center information was available for each test taker and was accurate. The other consideration was that prior studies (Lee & Haberman, 2013, 2018) had found regional effects on test taker performance. Because 100 cluster-based groups were needed per test form, the average jackknife group sizes, N avg , should be approximately 46, 30, and 34 for Forms 1, 2, and 3, respectively (see section "Data Source" for the three sample sizes).…”

Section: Constructing 100 Cluster-based Groupsmentioning

confidence: 99%

“…Either a single test center or an aggregate of test centers from the same geographic region can be treated as a cluster in these assessment samples. Previous studies (Lee & Haberman, , ) have found that test takers' performance on English proficiency assessments is related to geographic regions defined by the country or region in which test takers took the test (called test center country in this report). Lee and Haberman () found that predictors for regional effects explained more variation in the mean scores of the administrations than did other predictors, such as seasonality.…”

mentioning

confidence: 90%

“…Lee and Haberman () found that predictors for regional effects explained more variation in the mean scores of the administrations than did other predictors, such as seasonality. With different data sets from the same test, Lee and Haberman () showed that test takers' geographic region, defined either by test center country or by both test center country and native country (test takers' self‐reported country of birth), was the most important source of variation in test takers' scores compared to other background variables, such as gender and age. It is clear that test takers' performance on such an English test tended to vary from region to region.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Grouping Effects on Jackknifed Variance Estimation for Item Response Theory Scaling and Equating With Cluster‐Based Assessment Data

Wang

Qian

Lee

2018

Self Cite

Educational assessment data are often collected from a set of test centers across various geographic regions, and therefore the data samples contain clusters. Such cluster‐based data may result in clustering effects in variance estimation. However, in many grouped jackknife variance estimation applications, jackknife groups are often formed by a random grouping method that ignores the cluster structures of the data. In this study, we constructed both random and cluster‐based jackknife groups for data known to have cluster structures and compared the jackknifed standard errors, yielded by two different grouping methods, of item response theory (IRT) scaling coefficient estimates and equated scores. Three independent data samples from an international test of English were used for the study. The cluster‐based jackknife group results showed relatively larger jackknifed standard errors of scaling coefficient estimates and scale scores than the results of the random jackknife groups for all three data samples. For cluster‐based assessment data, the cluster‐based jackknife approach provides a more appropriate way to estimate the standard errors of the parameters of IRT calibration, scaling, and equating analyses.

Charting the Future of Assessments

Kyllonen,

Sevak,

Ober

et al. 2024

Assessment refers to a broad array of approaches for measuring or evaluating a person's (or group of persons') skills, behaviors, dispositions, or other attributes. Assessments range from standardized tests used in admissions, employee selection, licensure examinations, and domestic and international large‐scale assessments of cognitive and behavioral skills to formative K–12 classroom curricular assessments. The various types of assessments are used for a wide variety of purposes, but they also have many common elements, such as standards for their reliability, validity, and fairness—even classroom assessments have standards. We believe the future of assessment will involve a shift in emphasis on what skills will be measured, innovations in how we go about measuring them, the use of advanced technologies for test operations, and an expansion in the value and kinds of information that test takers will receive from taking the assessment.In this paper, we argue and provide evidence for our belief that the future of assessment contains challenges but is promising. The challenges include risks associated with security and exposure of personal data, test score bias, and inappropriate test uses, all of which may be exacerbated by the growing infiltration of artificial intelligence (AI) into our lives. The promise is increasing opportunities for testing to help individuals achieve their education and career goals and contribute to well‐being and overall quality of life. To help achieve this promise we focus on the evidence‐based science of measurement in education and workplace learning, a theme throughout this paper.