An Investigation of Undefined Cut Scores With the Hofstee Standard‐Setting Method

Abstract

This article provides an overview of the Hofstee standard‐setting method and illustrates several situations where the Hofstee method will produce undefined cut scores. The situations where the cut scores will be undefined involve cases where the line segment derived from the Hofstee ratings does not intersect the score distribution curve based on actual exam performance data. Data from 15 standard settings performed by a credentialing organization are used to investigate how common undefined cut scores are with the Hofstee method and to compare cut scores derived from the Hofstee method with those from the Beuk method. Results suggest that when Hofstee cut scores exist that the Hofstee and Beuk methods often yield fairly similar results. However, it is shown that undefined Hofstee cut scores did occur in a few situations. When Hofstee cut scores are undefined, it is suggested that one extend the Hofstee line segment so that it intersects the score distribution curve to estimate cut scores. Analyses show that extending the line segment to estimate cut scores often yields similar results to the Beuk method. The article concludes with a discussion of what these results may imply for people who want to employ the Hofstee method.

The European Certificate in Nephrology: towards harmonization and excellence in training

“…The pass mark is derived by the modified Angoff method [14, 15], which estimates what proportion of just passing candidates will answer the question correctly. A post hoc correction of the pass mark is applied using the Hofstee compromise [16] and then the final pass mark is calculated. For more details visit https://www.mrcpuk.org/mrcpukexaminations/results/exam-pass-marks.…”

Designing and developing an app to perform Hofstee cut-off calculations

“…1 A range of methods can be used to determine this point and are covered in several other papers. [2][3][4] In summary, however, most methods fall into three categories: norm-referenced (determined by the performance of the student group), criterion-referenced (pre-determined as an absolute cut-off point) and compromise methods (a compromise between the previous two methods is found). 4…”

A Critical Look into the Beuk Standard‐Setting Method

“…Among the methods for performing standard setting, compromise methods are some of easiest methods to explain to panelists and they are among the quickest methods to implement. The fact that compromise methods are easy to explain and implement makes them an attractive option to evaluate or adjust cut scores, or as a secondary method that can be performed following a test‐centered or examinee‐centered approach (see Geisinger, 1991; Mills & Melican, 1987, 1988; Wyse & Babcock, 2017). In fact, a common pairing of standard‐setting methods in credentialing programs is to use either the modified Angoff or Bookmark method along with the Hofstee and/or Beuk methods (see The American Board of Family Medicine, 2011; National Board of Osteopathic Medical Examiners, 2016; O'Neill, Marks & Reynolds, 2005; Wendt & Kenny, 2007).…”

“…Wyse and Babcock (2017) further explored undefined cut scores with the Hofstee method using data from 15 credentialing programs. They found that undefined cut scores were not a rare occurrence and that the Hofstee and Beuk methods tended to produce similar cut scores.…”

“…Despite the common use of the Hofstee and Beuk methods, a limited amount of research has explored the properties of these methods or has compared results from these methods with other methods. Some notable exceptions include Bowers and Shindoll (1989), Fabrey and Raymond (1987), Mills and Melican (1987), Wyse and Babcock (2017), and Wyse (in press). Bowers and Shindoll (1989) compared Hofstee and Beuk cut scores with modified Angoff cut scores for a 200‐item credentialing exam.…”