Certainty and safe consequence responses provide additional information from multiple choice question assessments

Abstract

BackgroundClinicians making decisions require the ability to self-monitor and evaluate their certainty of being correct while being mindful of the potential consequences of alternative actions. For clinical students, this ability could be inferred from their responses to multiple-choice questions (MCQ) by recording their certainty in correctness and avoidance of options that are potentially unsafe.MethodsResponse certainty was assessed for fifth year medical students (n = 330) during a summative MCQ examination by having students indicate their certainty in each response they gave on the exam. Incorrect responses were classified as to their inherent level of safeness by an expert panel (response consequence). Analyses compared response certainty, response consequence across student performance groupings.ResultsAs students’ certainty in responses increased, the odds they answered correctly increased and the odds of giving unsafe answers decreased. However, from some ability groups the odds of an incorrect response being unsafe increased with high certainty.ConclusionsCertainty in, and safeness of, MCQ responses can provide additional information to the traditional measure of a number correct. In this sample, even students below standard demonstrated appropriate certainty. However, apart from those scoring lowest, student’s incorrect responses were more likely to be unsafe when they expressed high certainty. These findings suggest that measures of certainty and consequence are somewhat independent of the number of correct responses to MCQs and could provide useful extra information particularly for those close to the pass-fail threshold.

Defining and tracking medical student self-monitoring using multiple-choice question item certainty

“…The first criterion is based on correctness increasing as levels of certainty increase [27,33,[38][39][40][41]. An alternative analysis could have been increasing certainty with increasing levels of correctness [28][29][30][31][32].…”

“…This does introduce a degree of peer-referencing within the definition of adequate self-monitoring, however the likelihood of being correct for answers given with high certainty has consistently been in the 80-90% range over several student groups, with different year groups and cohorts [27,[39][40][41] and was 84% in this study. One additional criterion for adequacy of selfmonitoring might be the odds of unsafe responses amongst the high certainty responses [27], as these would result in errors with the greatest impact in clinical practice. Adding the potential safety of responses to item level certainty [30][31][32] has been included in other research programmes.…”

“…However, these studies used certainty based on a probability correct estimate, such as “there is a 70% probability that this answer is correct”. This absolute measure of accuracy [ 17 ] is not authentic to clinical practice [ 27 ], as clinicians do not generally consider their likelihood of the clinical decision being correct in terms of probability, but rather ask (or should ask) “Do I need to check this? Am I about to do the right thing?” [ 1 – 4 ].…”

“…When considering self-monitoring as the degree of correctness for levels of certainty, we have used the conceptual description that frames certainty as the need to “look it up” [ 1 , 2 ] or “refer this problem to another individual” [ 1 ]. We started work in this area by recording correctness for levels of certainty in multiple choice question (MCQ) tests in research settings and in lower and higher stakes assessments, and during this developed certainty rating scale descriptors [ 27 , 38 – 41 ] that are used in this study (Table 1 ).…”

“…The rationale for the first criterion built on the assumption that an individual's self-monitoring would be demonstrated if their odds of responses to test items being correct increased as their levels of certainty in those responses increased [27,40], where questions answered with higher certainty were more likely to be answered correctly than questions answered with lower certainty [40]. Our first criterion was therefore defined as a statistically significant trend for increasing proportions correct with increasing levels of certainty.…”