When Should I Use a Measure to Support Instructional Improvement at Scale? The Importance of Considering Both Intended and Actual Use in Validity Arguments

Abstract

Despite the ease of accessing a wide range of measures, little attention is given to validity arguments when considering whether to use the measure for a new purpose or in a different context. Making a validity argument has historically focused on the intended interpretation and use. There has been a press to consider both the intended and actual interpretations and how users make sense of the data when constructing validity arguments, but the practice is not widespread. This paper contributes to existing research on validity by highlighting the value of attending to the actual interpretation and use of a measure aimed at supporting instructional improvement in mathematics. We describe the use of the same measure across two contexts to highlight the importance of attending to characteristics of both users and the contexts in which the measures are used when assessing the validity of inferences for the purpose of instructional improvement efforts.

Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability, Validity, and Feasibility

“…Much more research is needed to investigate specific validity arguments for these scores, for specific uses and contexts. Of particular interest are differences in the requirements for high-stakes summative inferences and uses (e.g., certification or evaluation), compared with lower-stakes formative uses (see, for example, Ing et al, 2020).…”