Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence

Abstract

Despite a lack of face validity, there continues to be heavy reliance on objective paper-and-pencil measures of clinical competence. Among these measures, the most common item formats are patient management problems (PMPs) and three types of multiple choice questions (MCQs): one-best-answer (A-types); matching questions (M-types); and multiple true/false questions (X-types). The purpose of this study is to compare the reliability, validity and efficiency of these item formats with particular focus on whether MCQs and PMPs measure different aspects of clinical competence. Analyses revealed reliabilities of 0.72 or better for all item formats; the MCQ formats were most reliable. Similarly, efficiency analyses (reliability per unit of testing time) demonstrated the superiority of MCQs. Evidence for validity obtained through correlations of both programme directors' ratings and criterion group membership with item format scores also favoured MCQs. More important, however, is whether MCQs and PMPs measure the same or different aspects of clinical competence. Regression analyses of the scores on the validity measures (programme directors' ratings and criterion group membership) indicated that MCQs and PMPs seem to be measuring predominantly the same thing. MCQs contribute a small unique variance component over and above PMPs, while PMPs make the smallest unique contribution. As a whole, these results indicate that MCQs are more efficient, reliable and valid than PMPs.

Impact of an Evidence-Based Medicine Curriculum on Resident Use of Electronic Resources: A Randomized Controlled Study

“…[10][11][12][13][14][15] The first session focused on formulating clinical questions using the patient, intervention, comparison, outcome method. 15 17 and have been shown to correlate with assessment by unannounced standardized patients, an accepted "gold standard." 18 We selected 15 multiple-choice questions based on clinical vignettes from the Medical Knowledge Self-Assessment Program questions CD.…”

Assessment formats in dental medicine: An overview

“…Classified as high [9], [11]A minimum of 40 high-quality questions are needed to yield α Cronbach’s α of 0.8 [6]. …”

Item Type and Cognitive Ability Measured: The Validity Evidence for Multiple True-False Items in Medical Specialty Certification

“…However, few studies of criterion-related validity have been reported (Albanese, Kent, & Whitney, 1979;Norcini, Swanson, Grosso, & Webster, 1985). Further, most of this research has used data from elementary, secondary, and undergraduate students; few studies have been reported using postgraduate professional examination data.…”