Response Processes in the Context of Validity: Setting the Stage

Hubley, Anita M.; Zumbo, Bruno D.

doi:10.1007/978-3-319-56129-5_1

Cited by 34 publications

(32 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As for what counts for response process, Hubley and Zumbo (2017) recently offered a broad scope that embraces not only individuals' cognition but also their behaviors and emotions. Hopefully, this more inclusive view could widen the variety of possible validity evidence based on response process.…”

Section: Response Process-based Validationmentioning

confidence: 99%

Linking Test-Taking Process to Performance Through Mixed-Effects Regression Models: A Response Process–Based Validation Study

Chen

Liu

2019

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

Answering the call for response process–based validation, this study shows how researchers can evaluate validity evidence of test scores based on the mental “processes” test-takers use, rather than based on correlations with other “outcome” measures. The proposed methods for process-based validation studies are demonstrated using a sample of 189 adults who took two listening comprehension tasks. Immediately after completing each task, the test-takers filled out a 10-item survey to reflect on the mental processes involved in reaching their answers. These 10 process variables attempted to capture five desired and five undesired response processes in answering multiple-choice listening comprehension questions. We investigated the relationships between these process variables and the binary outcome of item score (correct vs. incorrect) using mixed-effects logistic regression models, and showed how the results could provide validity evidence (or lack thereof). By doing so, we offer an alternative approach to study response process and test performance and encourage more process-based validation studies.

show abstract

Section: Response Process-based Validationmentioning

confidence: 99%

Linking Test-Taking Process to Performance Through Mixed-Effects Regression Models: A Response Process–Based Validation Study

Chen

Liu

2019

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

show abstract

“…American Psychological Association, and the National Council on Measurement in Education, 2014 ). Hubley and Zumbo ( 2017 ) defined response processes as “the mechanisms that underlie what people do, think, or feel when interacting with, and responding to, the item or task and are responsible for generating observed test score variation” (Hubley and Zumbo, 2017 , p. 2). Techniques of cognitive interviewing such as think-aloud and verbal probing procedures are well suited to elucidate response processes and provide evidence concerning the fit between observed response processes and the intended construct (Willis, 2005 ; Castillo-Díaz and Padilla, 2013 ; Padilla and Benítez, 2014 ; Launeanu, 2016 ).…”

Section: Introductionmentioning

confidence: 99%

Evaluation of Response Processes to the Danish Version of the Dutch Multifactor Fatigue Scale in Stroke Using the Three-Step Test-Interview

Cour

Norup

Schow

et al. 2021

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Validated self-report measures of post-stroke fatigue are lacking. The Dutch Multifactor Fatigue Scale (DMFS) was translated into Danish, and response process evidence of validity was evaluated. DMFS consists of 38 Likert-rated items distributed on five subscales: Impact of fatigue (11 items), Signs and direct consequences of fatigue (9), Mental fatigue (7), Physical fatigue (6), and Coping with fatigue (5). Response processes to DMFS were investigated using a Three-Step Test-Interview (TSTI) protocol, and data were analyzed using Framework Analysis. Response processes were indexed on the following categories: (i) “congruent,” response processes were related to the subscale construct; (ii) “incongruent,” response processes were not related to the subscale construct; (iii) “ambiguous,” response processes were both congruent and incongruent or insufficient to evaluate congruency; and (iv) “confused,” participants did not understand the item. Nine adults were recruited consecutively 10–34 months post-stroke (median = 26.5) at an outpatient brain injury rehabilitation center in 2019 [five females, mean age = 55 years (SD = 6.3)]. Problematic items were defined as <50% of response processes being congruent with the intended construct. Of the 38 items, five problematic items were identified, including four items of Physical fatigue and one of Mental fatigue. In addition, seven items posed various response difficulties to some participants due to syntactic complexity, vague terms, a presupposition, and a double-barrelled statement. In conclusion, findings elucidate the interpretative processes involved in responding to DMFS post-stroke, strengthen the evidence base of validity, and guide revisions to mitigate potential problems in item performance.

show abstract

“…• Post-assessment questionnaires and administrator reports (Eklöf and Knekta, 2017 [32]; Eklöf and Hopfenbeck, 2019 [33]; Hopfenbeck and Kjaernsli, 2016 [34]);…”

Section: Figure 1: Collecting Data On Response Processesmentioning

confidence: 99%

The uses of process data in large-scale educational assessments

2023

OECD Education Working Papers

View full text Add to dashboard Cite

The digital transition in educational testing has introduced many new opportunities for technology to enhance large-scale assessments. These include the potential to collect and use log data on test-taker response processes routinely, and on a large scale. Process data has long been recognised as a valuable source of validation evidence in assessments. However, it is now being used for multiple purposes across the assessment cycle. Process data is being deliberately captured and used in large-scale, standardized assessmentsmoving from viewing it as a 'by-product' of digital assessment, to its use 'by design' to extend understanding of test-taker performance and engagement. While these techniques offer significant benefits, they also require appropriate validation practices to ensure that their use supports reliable inferences and does not introduce unintended negative consequences. EDU/WKP(2023)1  5 THE USES OF PROCESS DATA IN LARGE-SCALE EDUCATIONAL ASSESSMENTS Unclassified Table of contentsAcknowledgements .

show abstract

Response Processes in the Context of Validity: Setting the Stage

Cited by 34 publications

References 38 publications

Linking Test-Taking Process to Performance Through Mixed-Effects Regression Models: A Response Process–Based Validation Study

Linking Test-Taking Process to Performance Through Mixed-Effects Regression Models: A Response Process–Based Validation Study

Evaluation of Response Processes to the Danish Version of the Dutch Multifactor Fatigue Scale in Stroke Using the Three-Step Test-Interview

The uses of process data in large-scale educational assessments

Contact Info

Product

Resources

About