Performance on the American Board of Family Medicine (ABFM) Certification Examination: Are Superior Test-Taking Skills Alone Sufficient to Pass?

Abstract

Introduction: Certification examinations used by American specialty boards have been the sine qua non for demonstrating the knowledge sufficient for attainment of board certification in the United States for more than 75 years. Some people contend that the examination is predominantly a test of superior test-taking skills rather than of family medicine decision-making ability. In an effort to explore the validity of this assertion, we administered the American Board of Family Medicine (ABFM) Certification to examinees who had demonstrated proficiency in taking standardized tests but had limited medical knowledge.Methods: Four nonphysician experts in the field of measurement and testing were administered one version of the 2009 ABFM certification examination. Scaled scores were calculated for each examinee, and psychometric analyses were performed on the examinees responses to examination items and compared with the performance of physicians who took the same examination.Results: The minimum passing threshold for the examination was a scaled score of 390, corresponding to 57.7% to 61.0% of questions answered correctly, depending on the version of the examination. The 4 nonphysician examinees performed poorly, with scaled scores that ranged from 20 to 160 (mean, 87.5; SD, 57.4). The number of questions answered correctly ranged from 24.0% to 35.1% (mean, 29.2%; SD, 0.05%). Rasch analyses of the examination items revealed that the nonphysician examinees were more likely to use guessing strategies in an effort to answer questions correctly. Distracter analysis suggest near-complete randomness in the nonphysician responses.Conclusions: Though all 4 nonphysician examinees performed better than would have been predicted by chance alone, none performed well enough to even fall within 8 SE below the passing thresholds; their performance was far below that of almost all physicians who completed the examination. Given that the nonphysicians relied heavily on the identifying cues in the phrasing of items and the manner in which response options were presented, the results affirm the notion that the ABFM certification examination is not primarily a measure of generic test-taking ability but measures information critical to the estimation of a family physician's knowledge sufficient for certification. Item analysis confirmed that items were well written, provided minimal cueing, and required medical knowledge to answer correctly. (J Am Board Fam Med 2011;24:175-180.)

Guest Family Physician Commentaries

“…5 The study concluded that, although all the participants seem to have performed better than chance would predict, the results affirm the notion that the ABFM board examination is not predominantly a measure of generic test-taking ability and clearly requires medical training to pass. As my father is fond of saying, "failing to plan is like planning to fail."…”

“…O'Neill et al 5 evaluated whether it was test-taking skills or physician ability that determined a passing score on the ABFM board certification examination. The study was designed using 4 nonphysicians who are considered experts in the field of certification and licensure testing.…”

Delaying the American Board of Emergency Medicine Qualifying Examination Is Associated With Poorer Performance

“…31 Simply possessing excellent test-taking skills is insufficient to pass a medical specialty board examination; medical knowledge is essential. 32 Some EM residency graduates delay taking the qualifying examination because they pursue fellowship training (ranging in length from 1 to 3 years). Our study was not designed to identify the experience of these individuals.…”

Pass Rates on the American Board of Family Medicine Certification Exam by Residency Location and Size

“…2 Through item analysis, a recent study confirmed that the ABFM CE test questions were well written, provided minimal cueing, and required specific medical knowledge as opposed to a generalized test-taking ability to answer questions correctly. 3 To improve performance on the examination, applicants are encouraged to increase their fund of medical knowledge and decision-making ability in clinical scenarios. 4 The Accreditation Council for Graduate Medical Education (ACGME) sets standards for program performance on the ABFM CE.…”