Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

Abstract

Background A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors’ background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements.

Examiners’ decision‐making processes in observation‐based clinical examinations

“…This study provides rich detail regarding the pragmatic and affective considerations that examiners employ when assessing observed clinical performance at an OSCE station. The finding that examiner decision making is a substantial cognitive task is consistent with results from other studies 40,47‐51,67 . This study delves deeper into the cognitive strategies that examiners use when judging clinical performance in OSCEs, providing both qualitative and quantitative data.…”

“…Previous exploratory studies on examiner cognition have focused on workplace‐based assessments (WBA) 40,47‐51 . Given the prominence of the OSCE format in medical education, it is important to explore examiner cognition in the context of time‐limited, multi‐station assessments.…”

The Scientific Rigor of the Objective Structured Examination for Competency Assessment in Health Sciences Education: A Systematic Review

“…The agreement in scores was higher for behavioral/professional and cognitive domains (8% variance of between-student scores 25 and intracorrelation coefficient [ICC], 0.39–1.0 30 ) than for the psychomotor domain (30% variance in between-student scores 25 and ICC, 0.25–0.63 30 ). Moreover, Roduta-Roberts et al 33 reported higher agreement in scores for the behavioral/professional (r-value, 0.59) than for the cognitive (r-value, 0.54–0.71) domain. Similarly, the correlation between the domain and holistic scores was also higher for the behavioral/professional (0.62–0.69) than for the cognitive (0.57–0.71) domain.…”

“…However, the effect of these factors was evaluated in only a few studies and, therefore, only a descriptive analysis of their effects on the internal construct of the OSCE was possible. The correlation between domain and holistic scores varied across different domains, from r = 0.36 24 for knowledge/interpretation stations to r = 0.69 33 and 0.79 24 for communication, with domain-specific effects contributing approximately 32–38% of the variance in between-student scores. 33 Regarding the type of stations used, Gorman et al 24 reported a higher correlation between domain and holistic scores for patient interaction stations (r ≥ 0.35) than for knowledge/clinical reasoning (written/video) stations (r = 0.176–0.77).…”

“…The correlation between domain and holistic scores varied across different domains, from r = 0.36 24 for knowledge/interpretation stations to r = 0.69 33 and 0.79 24 for communication, with domain-specific effects contributing approximately 32–38% of the variance in between-student scores. 33 Regarding the type of stations used, Gorman et al 24 reported a higher correlation between domain and holistic scores for patient interaction stations (r ≥ 0.35) than for knowledge/clinical reasoning (written/video) stations (r = 0.176–0.77). The time per station also influenced outcomes, 36 with decreased variability in student scores between stations when the time for transition and reading between stations was increased from 1 minute to 2 minutes.…”