The uses of process data in large-scale educational assessments

Abstract

The digital transition in educational testing has introduced many new opportunities for technology to enhance large-scale assessments. These include the potential to collect and use log data on test-taker response processes routinely, and on a large scale. Process data has long been recognised as a valuable source of validation evidence in assessments. However, it is now being used for multiple purposes across the assessment cycle. Process data is being deliberately captured and used in large-scale, standardized assessmentsmoving from viewing it as a 'by-product' of digital assessment, to its use 'by design' to extend understanding of test-taker performance and engagement. While these techniques offer significant benefits, they also require appropriate validation practices to ensure that their use supports reliable inferences and does not introduce unintended negative consequences. EDU/WKP(2023)1  5 THE USES OF PROCESS DATA IN LARGE-SCALE EDUCATIONAL ASSESSMENTS Unclassified Table of contentsAcknowledgements .

Process Data in Computer-Based Assessment

“…General definitions of process data encompass a range of data types, including, for example, verbal protocols, repeated survey questions, observational data, or physiological measures (see, e.g., Maddox, 2017, 2023; Oranje et al, 2017). Since the mid-twentieth century, process data in this broader sense have gained attention following the early work of researchers such as Lindquist (1951), Ryans and Frederiksen (1951), and Cronbach and Meehl (1955) in the course of validity discussions (see, e.g., Ercikan et al, 2020; Zumbo et al, 2023).…”

“…Thus, it is often unclear how observable actions can be indicative of construct-relevant mental processes. A multitude of different data sources can theoretically be employed in various constellations, building specific indicators and statistical indices (see, e.g., Maddox, 2023; Oranje et al, 2017). How to support the validity arguments of process indicators is, therefore, the center of discussions in the literature and the papers on this special issue (e.g., Arslan et al., 2023; Drake et al, 2023; Hahnel et al, 2023; Zumbo et al, 2023).…”

“…Collecting process data can be perceived as invasive as they may reflect sensitive and intimate information of an individual, such as their effort and struggle while answering items and solving problems (Maddox, 2023). Researchers and practitioners need to be aware of potential risks of misuse and comply with ethical guidelines and local regulations, such as the General Data Protection Regulation (GDPR) in Europe when collecting, storing, and analyzing (process) data.…”

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

“…Such information is recorded and collected via external instruments and encompasses diverse types of data, such as eye-tracking data, paradata (e.g., mouse clicks) or anthropological data (e.g., gestures; Hubley & Zumbo, 2017). Process data have recently been spotlighted, as technology-based assessments have advanced with the growth of data science and computational psychometrics, thereby increasing the opportunities for their exploitation across the entire assessment cycle (Goldhammer & Zehner, 2017;Maddox, 2023).…”

“…As a steppingstone, we hope the results of this study will be helpful for clarifying the behaviors of (un)successful participants in ColPS and will thus be conducive to the development of appropriate interventions (Greiff et al, 2018;Hao & Mislevy, 2019;Hickendorff et al, 2018;Teig et al, 2020). Furthermore, as we identified subgroups on the basis of the process data, the subgroups will be used to design better task situations and assessment tools in terms of validity and statistical scoring rules in the future (AERA, APA, & NCME, 2014;Goldhammer et al, 2020Goldhammer et al, , 2021Herborn et al, 2017;Hubley & Zumbo, 2017;Li et al, 2017;Maddox, 2023;von Davier & Halpin, 2013).…”