A Simulation Study to Compare Nonequivalent Groups With Anchor Test Equating and Pseudo‐Equivalent Group Linking

Equating the scores from different forms of a test requires collecting data that link the forms. Problems arise when the test forms to be linked are given to groups that are not equivalent and the forms share no common items by which to measure or adjust for this group nonequivalence. We compared three approaches to adjusting for group nonequivalence in a situation where not only is randomization questionable, but the number of common items is small. Group adjustment through either subgroup weighting, a weak anchor, or a mix of both was evaluated in terms of linking accuracy using a resampling approach. We used data from a single test form to create two research forms for which the equating relationship was known. The results showed that both subgroup weighting and weak anchor approaches produced nearly equivalent linking results when group equivalence was not met. Direct (random groups) linking methods produced the least accurate result due to nontrivial bias. Use of subgroup weighting and linking using the anchor test only marginally improved linking accuracy compared to using the weak anchor alone when the degree of group nonequivalence was small.

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

Comparisons Among Approaches to Link Tests Using Random Samples Selected Under Suboptimal Conditions

Kim¹,

Walker²

2021

“…The finding that weighted equating led to better equating results than the unweighted equating were consistent with other research on the use of common‐item equating with covariates (i.e., background variables correlated with the test scores). For example, Bränberg and Wiberg (2011) showed that the use of covariates such as gender and education with linear equating had less equating error than the regular linear equating without the use of the covariates; Sansivieri and Wiberg (2017) showed reduced equating error by using covariates in the IRT observed score equating context; and Lu and Guo (2018) showed that the use of the minimum discriminant information method to create equivalent groups along with the NEAT design had less equating error than the equating without the additional background information.…”

Section: Summary and Discussionmentioning

confidence: 99%

“…With an established testing program that uses common items for equating, he verified that PEG linking produces results similar to the conventional equatings with anchor tests. As an extension of the PEG modeling, Lu and Guo (2018) further proposed to use both background information and anchor test scores to adjust group ability difference. With simulation studies based on real data, they showed that the equating based on common items could be improved by incorporating the PEG adjustment procedure into the NEAT process.…”

mentioning

confidence: 99%

Effect of Statistically Matching Equating Samples for Common‐Item Equating

Kim

2021

Self Cite

This study evaluated the impact of subgroup weighting for equating through a common‐item anchor. We used data from a single test form to create two research forms for which the equating relationship was known. The results showed that equating was most accurate when the new form and reference form samples were weighted to be similar to the target population. When the target population was a combination of the two equating samples and one sample was weighted to be similar to the other, the equating was less accurate but still much more accurate than equating with unweighted samples.

“…Several psychometricians at ETS conducted research to assess the practical implications of PEG for large‐scale assessment programs using either real or simulated data (see Kim & Lu, 2018; Lu & Guo, 2018; Oh et al, 2015; Xi et al, 2015). A recent empirical investigation (Kim, 2020) using certification tests demonstrated how the PEG approach could be used to explore potential mode effects across RP and TC delivery.…”

mentioning

confidence: 99%

Assessing Mode Effects of At‐Home Testing Without a Randomized Trial

Kim¹,

Walker²

2021

In this investigation, we used real data to assess potential differential effects associated with taking a test in a test center (TC) versus testing at home using remote proctoring (RP). We used a pseudo‐equivalent groups (PEG) approach to examine group equivalence at the item level and the total score level. If our assumption holds that the PEG approach removes between‐group ability differences (as measured by the test) reasonably well, then a plausible explanation for any systematic differences in performance between TC and RP groups that remain after applying the PEG approach would be the operation of test mode effects. At the item level, we compared item difficulties estimated using the PEG approach (i.e., adjusting only for ability differences between groups) to those estimated via delta equating (i.e., adjusting for any systematic differences between groups). All tests used in this investigation showed small, nonsystematic differences, providing evidence of trivial effects associated with at‐home testing. At the total score level, we linked the RP group scores to the TC group scores after adjusting for group differences using demographic covariates. We then compared the resulting RP group conversion to the original TC group conversion (the criterion in this study). The magnitude of differences between the RP conversion and the TC conversion was small, leading to the same pass/fail decision for most RP examinees. The present analyses seem to suggest little to no mode effects for the tests used in this investigation.