Historical Perspectives on Score Comparability Issues Raised by Innovations in Testing

Abstract

While score comparability across test forms typically relies on common (or randomly equivalent) examinees or items, innovations in item formats, test delivery, and efforts to extend the range of score interpretation may require a special data collection before examinees or items can be used in this way-or may be incompatible with common examinee or item designs altogether. When comparisons are necessary under these nonroutine conditions, forms still must be connected by something and this article focuses on these form-invariant connective somethings. A conceptual framework for thinking about the problem of score comparability in this way is given followed by a description of three classes of connectives. Examples from the history of innovations in testing are given for each class.

Recent Challenges to Maintaining Score Comparability: A Commentary

“…Whereas Baldwin and Clauser (2022) use innovative assessments as the source of their insights into score comparability, Puhan and Kim are motivated specifically by concerns about the comparability of scores obtained in controlled test centers (TC) versus scores obtained from remote‐proctored (RP) tests. They provide a fine overview of the challenges of remote proctoring and the procedures appropriate to deal with it.…”

“…The authors make a distinction, as we do, between data collection designs that assume a common population and designs that rely on common items, as well as those that rely on other types of common content. In addition, the article describes a subset of what Baldwin and Clauser (2022) call second-order connectives and which we just called third-order connectives, in more detail. In particular, they cite procedures developed by Livingston (2014) and Haberman (2015) for potential use in settings where neither common people nor common items are available.…”

“…In closing, the discussion of data collection designs differs somewhat from the treatment in Baldwin and Clauser (2022), and the authors illustrate how to check if the linking between two tests is invariant across modes.…”

“…In closing, the Moses article and the Baldwin and Clauser (2022) article can be viewed as complementary philosophical bookends to the three empirical papers between them. Whereas Baldwin and Clauser provide a new vocabulary for describing links between tests, Moses tries to stretch existing frameworks for thinking about linking to address increasingly divergent measurement conditions, and concludes that divergent measurement conditions necessitate the need to emphasize limited comparability.…”

“…We address the papers in the following order: Baldwin and Clauser; Puhan and Kim; Jones, Tong, Liu, Borglum and Primoli; Liu and Becker, Moses. Baldwin and Clauser (2022) provide a conceptual framework for thinking about the problem of score comparability in terms of what they call connectives and follow that with examples from the history of innovations in testing that employ different connectives. To address issues of comparability associated with remote proctoring of exams, Puhan and Kim (2022) summarize statistical procedures that can be used to evaluate potential mode effects for both individual items and total scores.…”