The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

Abstract

The decision to pass or fail a medical student is a 'high stakes' one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students' OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.

The Objective Borderline Method: a probabilistic method for standard setting

“…We would not revoke the cP of students who earned that even if they were within the range of uncertainty, since their ability might be at borderline or cP level but not cF. It is important to emphasise the use of the Pr(cP) and Pr(cF) probabilities for decision-making, but we do not suggest that those are probabilities of 'real events', thus they were originally described as 'conceptual' probabilities (Shulruf et al 2013).…”

“…(Sutcliffe 1958;Barker, Pistrang, and Elliott 2005). Based on this concept, a new approach to standard setting, the OBM, was introduced recently (Shulruf et al 2013). The OBM uses the probability that a student would pass two 'conceptual' tests in order to set up a pass/ fail cut-score.…”

“…This paper aims to build upon the approach outlined in the original paper (Shulruf et al 2013) with a fuller explanation and some minor refinements of the method, before going on to test, the OBM, using simulated data.…”

“…In other words, the grades' score ranges may not be mutually exclusive (Cizek and Bunch 2007a;Kramer et al 2003). For example, the borderline group and regression methods use only the test scores and global judgement, comparing test item scores deemed objective with pre-determined global judgments' categories deemed subjective, to predict what the pass/fail cut-score should be (Smee 2001;Wilkinson et al 2003;Boursicot, Roberts, and Pell 2007;Shulruf et al 2013). On the other hand, this might be advantageous as the examination scores and grades are established at one time by the examiners, and no additional post-examination judgement-related uncertainty is added to the standard setting process (Pant et al 2009;Muijtjens et al 2003).…”

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

“…[52] This method assumes that examiners, in the absence of a panel of judges, are proficient in classifying examinee performance as fail, borderline or pass and thus, can provide a provisional pass mark. Using a probabilistic model, the proportions of examinees falling into the above classes are used to determine an official pass mark for the relevant assessment.…”