Seeing the ‘black box’ differently: assessor cognition from three research perspectives

Abstract

Although each of the perspectives discussed in this paper advances our understanding of assessor cognition and its impact on WBA, every perspective has its limitations. Following a discussion of areas of concordance and discordance across the perspectives, we propose a coexistent view in which researchers and practitioners utilise aspects of all three perspectives with the goal of advancing assessment quality and ultimately improving patient care.

Managing extremes of assessor judgment within the OSCE

“…Small bodies of work have started to concentrate on identifying examiners that are at the extremes of marking at station level, revealing in one study that <0.3% of examiners can be classed as extreme [ three standard deviations beyond the mean of all raters] (Bartman et al 2013). Whilst the overall proportion in these case studies, it are a reflection of a complex environment which exerts powerful effects on assessment through test constructs and scoring formats and judgements about safe clinical care (Kogan et al 2014).…”

“…Candidates tend to perform better in repeat or supplementary assessments, and recent literature highlights that some of the difficulties in setting the standard for such an extreme subgroup, where students numbers may be smaller and individual assessor effects more pronounced [in this example, typically 30 students and 15-20 assessors] (McManus & Ludka 2012;Pell et al 2012;Homer et al 2015). There are likely to be multiple factors contributing to the variance seen in Case 3, including contrast effects between weaker assessors and practitioners are likely to exert powerful effects (Kogan et al 2014;Yeates 2015).…”

“…This work has explored the complex, dynamic interaction of performance , or through use of constructivist paradigms illustrating that differing assessor judgements reflect meaningful idiosyncrasy (Gingerich et al 2014). The constructivist approach to B s & Brooks 1993), and the role of the assessor is richer and multifaceted (Harden & Crosby 2000).…”

Proceed With Caution: Implementing Competency-Based Graduate Medical Education

“…Accumulating evidence suggests that no rating scale is likely to eliminate assessor disagreement. [17][18][19][20] One reason may be that assessors are not all evaluating the same aspect of performance. 20 Different assessors may be focusing on different aspects at different times with different learners.…”

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall

“…[1][2][3][4] Another source of variability appears to be the format of the rating instrument. Efforts to improve the reliability of WBAs have generally been directed toward manipulating forms and rating scales.…”