Performance of a dangerous answer subtest within a subspecialty certifying examination

Abstract

Previous studies have been reported suggesting that a population of candidates for specialty board certification can be identified who pass the certification examination but who give an unduly high number of 'dangerous' responses, indicating their acceptance of actively harmful actions. To confirm these results, a retrospective analysis was undertaken of a Subspecialty Board of Nephrology certification examination. Experts identified a subtest of 75 dangerous answers. The performance of candidates on this subtest was compared with their performance on the total examination. The subtest was moderately reliable, ranked criterion groups appropriately, had a correlation with the total test of 0.71, and using the standard used for the total examination, identified the certification status of 84% of candidates. However, when the correlation was corrected for unreliability, the correlation became 1.0, indicating that a dangerous answer subtest, at least in this examination and population, does not identify a unique population of certified but 'dangerous' doctors.

Certainty and safe consequence responses provide additional information from multiple choice question assessments

“…Replicating aspects of previous findings in different student groups increases the generalisability of the results. Previously, others reported that the number of unsafe responses is proportional to the numbers of incorrect responses, and therefore this additional metric adds nothing [ 25 , 27 ]. However, all previous investigations have been retrospective, where the candidates were not given any scoring disincentive for responding unsafely [ 25 – 27 ].…”

“…Concern has also been raised about those who have sufficiently good examination performance to score above a pass threshold in assessments but whose incorrect responses are considered dangerous, possibly demonstrating a lack of consideration of consequences of decisions [ 25 – 28 ]. Thus, getting enough correct responses to pass is not offset by noting their incorrect dangerous responses.…”

Are medical postgraduate certification processes valid? A systematic review of the published evidence

“…They then compared scores on the subset to composite scores, to find out whether any candidates who passed had `failed' the subset. In the American Board of Internal Medicine's Subspecialty Board of Nephrology, the 6·6% of certified candidates who `failed' the subset had no demographic differences to those who passed 69 . Further analysis of subset to composite score corrected for low reliabilities revealed that the subset did not measure unique properties.…”

“…Two of the included papers, both from the USA Specialty Boards, one nephrology, 69 the other anaesthetics, 63 investigated the impact of separately scoring subsets of written questions. The authors identified a subset of questions with answers that could, if selected incorrectly, be potentially dangerous to patients.…”

Does It Matter if You “Kill” the Patient or Order Too Many Tests? Scoring Alternatives for a Test of Clinical Reasoning Skill

“…For example, Webster, Goldfarb, Norcini, Shea, and Murray (1987) studied the relationship between the number of dangerous options selected on the American Board of Internal Medicine's Subspecialty Board of Nephrology multiple-choice examination and overall performance on the test. They found that most examinees selecting dangerous treatment options performed poorly on the rest of the test.…”