Judges' Use of Examinee Performance Data in an Angoff Standard‐Setting Exercise for a Medical Licensing Examination: An Experimental Study

Abstract

Although the Angoff procedure is among the most widely used standard setting procedures for tests comprising multiple‐choice items, research has shown that subject matter experts have considerable difficulty accurately making the required judgments in the absence of examinee performance data. Some authors have viewed the need to provide performance data as a fatal flaw for the procedure; others have considered it appropriate for experts to integrate performance data into their judgments but have been concerned that experts may rely too heavily on the data. There have, however, been relatively few studies examining how experts use the data. This article reports on two studies that examine how experts modify their judgments after reviewing data. In both studies, data for some items were accurate and data for other items had been manipulated. Judges in both studies substantially modified their judgments whether the data were accurate or not.

Non‐numeric Intrajudge Consistency Feedback in an Angoff Procedure

“…The effect observed in the current study cannot be attributed to what Clauser et al. () characterized as “mechanical” reliance on the provided conditional p ‐values. There were no recommended p ‐values or Angoff ratings in the feedback.…”

Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch

“…Returning to the Ebel method, it seems that at the item-level there have been problems in getting 'it right' (Homer et al, 2012;Clauser et al, 2008;Clauser et al, 2009;Clauser et al, 2014;Mee et al, 2013;Margolis et al, 2016), but through our work using Rasch alongside Ebel, we are now more confident that examiner judgements are appropriate, certainly across the tests as a whole.…”

“…There is compelling evidence that test-centred approaches to setting standards on medical education knowledge assessments are far from perfect (Clauser et al, 2008;Clauser et al, 2009;Clauser et al, 2014;Margolis and Clauser, 2014;Mee et al, 2013;Margolis et al, 2016;Shulruf et al, 2016). Experts are generally found to be poor at judging the difficulty of items, even with normative data.…”

“…Experts are generally found to be poor at judging the difficulty of items, even with normative data. For example, in an experimental study (Clauser et al, 2009), judges were not able to make consistent predictions about item difficulty, and when given normative data (i.e. item performance statistics) revised their judgments to more closely match this data, even when this data was randomly generated.…”

Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method

“…This suggests that judges seem to defer to the data in all cases, suggesting reliance on performance data may supersede content expertise. 20 One unanswered question from the study was the impact of providing performance data on the pass rate of the examination. This study found that judgment based on expert knowledge of content and learners can be overwhelmed by performance data, particularly when the initial score was less than 100% for an item.…”