Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

Abstract

Absolute standard setting procedures are recommended for assessment in medical education. Absolute, test-centred standard setting procedures were introduced for written assessments in the Liverpool MBChB in 2001. The modified Angoff and Ebel methods have been used for short answer question-based and extended matching question-based papers, respectively. Data collected has been analysed to investigate whether reliable standards can be achieved for small-scale, medical school-based assessments, to establish the minimum number of judges required and the effect of a discussion phase on reliability. The root mean squared error (RMSE) has been used as a measure of reliability and used to compute 95% confidence intervals for comparison to the examination statistics. The RMSE has been used to calculate the minimum number of judges required to obtain a predetermined minimum level of reliability, and the effect of the number of judges and number of items have been examined. Values of the RMSE obtained vary from 0.9 to 2.2%. Using average variances across each paper type, the minimum number of judges to obtain a RMSE of less than 2% is 10 or more judges before discussion or 6 or more judges after discussion. The results indicate that including a discussion phase improves the reliability and reduces the minimum number of judges required. Decision studies indicate that increasing the number of questions included in the assessments would not significantly improve the reliability of the standard setting.

Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality

“…With regard specifically to Ebel-based methods, there are studies that compare Ebel standard setting with other such methods (Downing et al 2003, Skakun & Kling 1980. There is also a recent study that investigates the impact of the number of examiners, and the extent of their deliberations, on the reliability of pass marks derived from Ebel judgments (Fowell et al 2006). However, there is little evidence of previous research into precisely how well examiner estimates of item difficulty under Ebel correspond with actual item performance.…”

An introduction to standard setting methods in dentistry

“…11,12 There is still no consensus as to the exact number of judges needed, and although previous studies have suggested a range of 5-20, most authors suggest that a group of 10 is an appropriate number. [16][17][18] Judges should also be knowledgeable of the curriculum that is being assessed, the abilities of the student cohort and should be ideally selected with a balanced mix of age, gender, educational experience and subject experience.…”

Food for Thought: A New Compromised Method of Standard Setting

“…One may argue that with only six judges in each panel, our standard set pass mark may have been deviated from the true pass score for the paper. It has been shown that the calibration and training of the judges can result in reduction of the number of the judges (Fowell et al, 2008) but the above argument still remains a possibility. When comparing the outcomes of the different methods, the May 2015 diet of exams for the BDS year 3 class showed the most significant discrepancy.…”

“…To add to the drawbacks of these methods one can argue that these methods are time consuming, requiring considerable time input and commitment from mostly senior academic members of staff. Literature recommends 10-15 judges per panel of examination as an optimal number (Fowell, Fewtrell, & McLaughlin, 2008;Hurtz & Hertz, 1999), which for most medical and dental schools this is a large stretch on their already scarce resources.…”