How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: use of an objective structured clinical examination (OSCE)

Abstract

Background: Assessing residents' understanding and application of the 6 intrinsic CanMEDS roles (communicator, professional, manager, collaborator, health advocate, scholar) is challenging for postgraduate medical educators. We hypothesized that an objective structured clinical examination (OSCE) designed to assess multiple intrinsic CanMEDS roles would be sufficiently reliable and valid. Methods:The OSCE comprised 6 10-minute stations, each testing 2 intrinsic roles using case-based scenarios (with or without the use of standardized patients). Residents were evaluated using 5-point scales and an overall performance rating at each station. Concurrent validity was sought by correlation with in-training evaluation reports (ITERs) from the last 12 months and an ordinal ranking created by program directors (PDs).Results: Twenty-five residents from postgraduate years (PGY) 0, 3 and 5 participated. The interstation reliability for total test scores (percent) was 0.87, while reliability for each of the communicator, collaborator, manager and professional roles was greater than 0.8. Total test scores, individual station scores and individual CanMEDS role scores all showed a significant effect by PGY level. Analysis of the PD rankings of intrinsic roles demonstrated a high correlation with the OSCE role scores. A correlation was seen between ITER and OSCE for the communicator role, while the ITER medical expert and total scores highly correlated with the communicator, manager and professional OSCE scores. Conclusion:An OSCE designed to assess the intrinsic CanMEDS roles was sufficiently valid and reliable for regular use in an orthopedic residency program.Contexte : Évaluer la compréhension et l'application des 6 rôles intrinsèques Can-MEDS (communicateur, professionnel, gestionnaire, collaborateur, promoteur de la santé, érudit) chez les résidents pose un défi pour les responsables de la formation médi-cale postdoctorale. Nous avons émis l'hypothèse selon laquelle un examen clinique objectif structuré (ECOS) conçu pour évaluer plusieurs rôles CanMEDS intrinsèques serait suffisamment fiable et valide.Méthodes : L'ECOS comportait 6 stations de 10 minutes, permettant chacune d'évaluer 2 rôles intrinsèques à l'aide de scénarios basés sur des cas (avec ou sans recours à des patients standardisés). Les résidents ont été notés au moyen d'échelles en 5 points et d'une évaluation globale de leur rendement à chacune des stations. La validité convergente a été vérifiée par corrélation avec les rapports d'évaluation en cours de formation (RÉF) des 12 mois précédents et un classement chiffré créé par les directeurs du programme (DP).Résultats: Vingt-cinq résidents des années 0, 3 et 5 y ont participé. La fiabilité interstation pour les scores totaux aux tests (en pourcentage) a été de 0,87, tandis que la fiabilité pour chacun des rôles de communicateur, collaborateur, gestionnaire et professionnel, a été supérieure à 0,8. Les scores totaux aux tests, les scores aux stations individuelles et les scores pour les rôles CanMEDS indivi...

The state of health advocacy training in postgraduate medical education: a scoping review

“…103 However, the current assessment frameworks that prioritise reliability, defensibility and standardisation may be incompatible with the intended goals of meso-and macro-level advocacy, and the collective representation and action necessary to effect social change and improve the health of communities and populations. 130 We agree with Martin et al 41 that common assessment parameters do not capture the end goal of producing engaged and committed health advocates. Rather, long-term health advocacy engagement using postgraduate career tracking can be a meaningful measure of success in health advocacy training.…”

“…This review uncovered a number of publications willing to name the elephant in the room – our collective discomfort with ‘activism’ – and embrace actions that ‘result in institutional, social, economic or political change’ . However, the current assessment frameworks that prioritise reliability, defensibility and standardisation may be incompatible with the intended goals of meso‐ and macro‐level advocacy, and the collective representation and action necessary to effect social change and improve the health of communities and populations . We agree with Martin et al .…”

El Examen Clínico Objetivo Estructurado (ECOE) en la evaluación de competencias de comunicación y profesionalismo en los programas de especialización en Medicina

“…Pueden ser actores o profesionales de salud que han sido entrenadas para la simulación, personas sanas, o bien personas enfermas que han sido entrenadas para estandarizar su propia condición ante las repetidas evaluaciones a las que será expuesta (23,35).…”

A Comparison of Assessment Tools: Is Direct Observation an Improvement Over Objective Structured Clinical Examinations for Communications Skills Evaluation?

“…A well-designed OSCE can assess communications skills and diagnostic capabilities across postgraduate years. 3,5,6 While the OSCE is a useful tool, it is a scheduled event that residents can prepare for by reviewing relevant topics prior to the examination. Thus, the assessment may not reflect resident performance in actual clinical situations.…”