ROC and Loss Function Analysis in Sequential Testing

Abstract

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods

“…which combination has minimum loss? The Loss function provides a tool for evaluating the classification accuracy of diagnostic tests (30–32). Loss function analyses the loss involved with incorrect decisions, i.e.…”

Sequential testing in high stakes OSCE: a stratified cross-validation approach

“…In this scenario the field of medical education has become increasingly valuable and clinical examinations have far-reaching implications for society. In the effort to develop cost-effective decision making for examination development a multitude of different methods has been proposed (Colliver et al, 1991, Colliver, Vu and Barrows, 1992, Cass et al, 1997, Currie, Selvaraj and Cleland, 2015, Smee et al, 2003, Muijtjens et al, 2000, Regehr and Colliver, 2003, Muijtjens, Van Luijk and Van Der Vleuten, 2006. Computational methods entered this debate as they constitute powerful and flexible tools for those test developers interested in transitioning to sequential testing.…”

“…Another consideration in choosing the appropriate number of stations in a screening test concerns the balance between (a) test classification errors (are borderline and non-borderline students correctly classified) and (b) the necessity to minimize the cost of delivery of the exam. Several studies have investigated the best methodology to determine the appropriate number of stations to include in a screening OSCE (Colliver et al, 1991, Colliver, Vu and Barrows, 1992, Cass et al, 1997, Currie, Selvaraj and Cleland, 2015, Smee et al, 2003, however currently no consensus has been reached (Muijtjens et al, 2000, Muijtjens, Van Luijk andVan Der Vleuten, 2006). Receiver operator characteristic (ROC) curves are the most prevalent statistical methodology utilised to model the predictive ability of the initial screening test to achieve appropriate certainty of the borderline decision (Regehr and Colliver, 2003).…”

The benefits of sequential testing: Improved diagnostic accuracy and better outcomes for failing students

“…Previous research (Currie and Cleland, 2016b;Currie et al, 2015;Jalili and Hejri, 2016;Muijtjens et al, 2006) has attempted to explore the 'quality' of sequential testing using whole test, largely psychometrically-driven, analyses (e.g. selecting stations to maximise reliability and decision-making).…”