The Validity of Multiple Choice Achievement Tests as Measures of Competence in Medicine

Levine, Harold G.; McGuire, Connor; Nattress, Leroy William

doi:10.3102/00028312007001069

Cited by 47 publications

(17 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…First, the observed discrepancy can be attributed in part to the structure of the items in the final test. As discussed above, multiple-choice items largely relate to the memorization of simple facts rather than to the desired analytical processing of knowledge associated with the development of rich knowledge representations (see, for example, Levine, McGuire, and Natress 1970;Frederiksen 1984;Birenbaum 1996). Therefore, even participants with low-quality knowledge representations might have been able to arrive at a satisfactory score on the final test.…”

Section: Discussionmentioning

confidence: 99%

Assessing Knowledge Structures in a Constructive Statistical Learning Environment

Verkoeijen

Imbos

Wiel

et al. 2002

Journal of Statistics Education

View full text Add to dashboard Cite

In this report, the method of free recall is put forward as a tool to evaluate a prototypical statistical learning environment. A number of students from the faculty of Health Sciences, Maastricht University, the Netherlands, were required to write down whatever they could remember of a statistics course in which they had participated. By means of examining the free recall protocols of the participants, insight can be obtained into the mental representations they had formed with respect to three statistical concepts. Quantitative as well as qualitative analyses of the free recall protocols showed that the effect of the constructive learning environment was not in line with the expectations. Despite small-group discussions on the statistical concepts, students appeared to have disappointingly low levels of conceptual understanding.

show abstract

Section: Discussionmentioning

confidence: 99%

Assessing Knowledge Structures in a Constructive Statistical Learning Environment

Verkoeijen

Imbos

Wiel

et al. 2002

Journal of Statistics Education

View full text Add to dashboard Cite

show abstract

“…The most commonly used of the paper‐and‐pencil methods are multiple choice questions (MCQs) and patient management problems (PMPs). Traditionally, it has been thought that MCQs measure knowledge, while PMPs, by virtue of the fact that they simulate a clinical encounter, measure knowledge plus problem‐solving skills (Levine et al , 1970; Levine, 1978). Studies investigating whether MCQs and PMPs measure different aspects of competence have produced variable results.…”

Section: Introductionmentioning

confidence: 99%

Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence

et al. 1985

View full text Add to dashboard Cite

Despite a lack of face validity, there continues to be heavy reliance on objective paper-and-pencil measures of clinical competence. Among these measures, the most common item formats are patient management problems (PMPs) and three types of multiple choice questions (MCQs): one-best-answer (A-types); matching questions (M-types); and multiple true/false questions (X-types). The purpose of this study is to compare the reliability, validity and efficiency of these item formats with particular focus on whether MCQs and PMPs measure different aspects of clinical competence. Analyses revealed reliabilities of 0.72 or better for all item formats; the MCQ formats were most reliable. Similarly, efficiency analyses (reliability per unit of testing time) demonstrated the superiority of MCQs. Evidence for validity obtained through correlations of both programme directors' ratings and criterion group membership with item format scores also favoured MCQs. More important, however, is whether MCQs and PMPs measure the same or different aspects of clinical competence. Regression analyses of the scores on the validity measures (programme directors' ratings and criterion group membership) indicated that MCQs and PMPs seem to be measuring predominantly the same thing. MCQs contribute a small unique variance component over and above PMPs, while PMPs make the smallest unique contribution. As a whole, these results indicate that MCQs are more efficient, reliable and valid than PMPs.

show abstract

“…Such questions enable the course content to be sampled widely, are reliable and easy to score. However, they suffer from poor validity (Levine et al 1970) and are difficult to construct in a manner which probes something more than recall of isolated fragments of factual knowledge. Over-emphasis on these questions may direct students' learning patterns away from other more important skills to rote learning and the acquisition of facts (Fredericksen 1984).…”

Section: Discussionmentioning

confidence: 99%

A multi-format Interdisciplinary Final Examination

Collins

Gamble

1996

Medical Education

View full text Add to dashboard Cite

Limitations of the traditional final medical examination for the assessment of clinical competence led to such developments as simulated patients and the Objective Structured Clinical Examination (OSCE). An interdisciplinary OSCE incorporating simulated patients and involving nine disciplines was introduced into the final examination in the Auckland School of Medicine to supplement the written papers and the long case. Six-hundred and eight students were assessed over a 6-year period. Each of the three examination modes provided good discriminatory power. Significant correlations were found between the tests, but this does not mean one or more is redundant. Principal component analysis showed that a single significant factor accounted for over half the variance in the final assessment. This factor was equally weighted to the three examinations. A variety of evaluative methods are necessary to assess a student's competence and greater emphasis should be placed on those methods which encourage the learning of clinical skills and concurrently provide an appropriate mechanism for assessing them. The changes introduced have been supported by students and teachers and have fostered the learning of important clinical skills. Efforts to standardize the single long case have not overcome the criticisms surrounding its use, particularly in summative assessment.

show abstract

The Validity of Multiple Choice Achievement Tests as Measures of Competence in Medicine

Cited by 47 publications

References 3 publications

Assessing Knowledge Structures in a Constructive Statistical Learning Environment

Assessing Knowledge Structures in a Constructive Statistical Learning Environment

Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence

A multi-format Interdisciplinary Final Examination

Contact Info

Product

Resources

About