Impact of integrating objective structured clinical examination into academic student assessment: Large-scale experience in a French medical school

Abstract

Purpose Objective structured clinical examinations (OSCE) evaluate clinical reasoning, communication skills, and interpersonal behavior during medical education. In France, clinical training has long relied on bedside clinical practice in academic hospitals. The need for a simulated teaching environment has recently emerged, due to the increasing number of students admitted to medical schools, and the necessity of objectively evaluating practical skills. This study aimed at investigating the relationships between OSCE grades and current evaluation modalities. Methods Three-hundred seventy-nine 4th-year students of University-of-Paris Medical School participated to the first large-scale OSCE at this institution, consisting in three OSCE stations (OSCE#1–3). OSCE#1 and #2 focused on cardiovascular clinical skills and competence, whereas OSCE#3 focused on relational skills while providing explanations before planned cholecystectomy. We investigated correlations of OSCE grades with multiple choice (MCQ)-based written examinations and evaluations of clinical skills and behavior (during hospital traineeships); OSCE grade distribution; and the impact of integrating OSCE grades into the current evaluation in terms of student ranking. Results The competence-oriented OSCE#1 and OSCE#2 grades correlated only with MCQ grades (r = 0.19, P<0.001) or traineeship skill grades (r = 0.17, P = 0.001), respectively, and not with traineeship behavior grades (P>0.75). Conversely, the behavior-oriented OSCE#3 grades correlated with traineeship skill and behavior grades (r = 0.19, P<0.001, and r = 0.12, P = 0.032), but not with MCQ grades (P = 0.09). The dispersion of OSCE grades was wider than for MCQ examinations (P<0.001). When OSCE grades were integrated to the final fourth-year grade with an incremental 10%, 20% or 40% coefficient, an increasing proportion of the 379 students had a ranking variation by ±50 ranks (P<0.001). This ranking change mainly affected students among the mid-50% of ranking. Conclusion This large-scale French experience showed that OSCE designed to assess a combination of clinical competence and behavioral skills, increases the discriminatory capacity of current evaluations modalities in French medical schools.

Objective structured clinical examination versus traditional written examinations: a prospective observational study

“…In our opinion, these results suggest that OSCE brings and added value to MCQs and other traditional examination modalities by evaluating a large variety of skills and in a different way: for instance, MCQs assess mainly medical knowledge and have little ability to assess clinical skills [ 23 ]. Consistently with other studies we observed a significantly larger distribution of grades obtained at OSCE compared to grades from other academic evaluation modalities [ 11 ]. This underlies the potential discriminating power of OSCE for student ranking.…”

“…This raises the question of the real added value of these evaluations. Matet et al showed that OSCE increased the discriminatory capacity of current evaluations modalities in French medical schools [ 11 ]. Furthermore, Pugh et al showed a correlation ranging from 0.305 ( p = 0.002) to 0.516 ( p < 0.001) between scores from an OSCE progress test and the written component of the national high-stakes examination [ 22 ].…”

“…Finally, the OSCE scores were significantly lower than those obtained with the other types of examination. Interestingly, in the study of Matet et al, this correlation was even weaker although significant, but the OSCE-scores were higher than those obtained at MCQ-based examinations [ 11 ]. Other studies also reported similar findings with a correlation coefficient between OSCE and MCQ ranging from 0.3 to 0.4 [ 12 , 16 ].…”

“…Problem-based learning and clinical simulation have been playing an increasingly important role in contemporary medical training [ 1 – 8 ]. For many years, objective structured clinical examination (OSCE) has shown its effectiveness in assessing medical students, and is already commonly used in many countries such as the United States, Canada, Australia or India; however, its integration in national medical training programs has been quite recent in many countries, especially in Europe [ 3 , 9 – 11 ].…”

Clinical experience can compensate for inferior academic achievements in an undergraduate objective structured clinical examination

“…Chan et al [12] tested if experiences in public speaking or performances (music, arts, drama) affected the results of an OSCE examination and found modest positive effects. Matet et al [13] showed for a French medical school that OSCE results in behaviouroriented stations (weakly) correlate with higher traineeship skills, while competence-oriented stations had only ties with performances in MC-exams. Kirton and Kravitz [14] concluded that OSCEs and MC-exams measure different competencies by explicitly targeting clinical skills.…”