A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents

Abstract

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

A model to teach concomitant patient communication during psychomotor skill development

“…This feedback can then be used to refocus the learner's goals for the next encounter, and foster the development of their selfassessment skills. With ongoing practice and facilitated reflection, skill acquisition can be scaffolded from initially using role-plays to then performing the skill on a simulated patient (Pugh et al 2015), and finally in the workplace. Teaching a communication skill culminates with the educator providing objective feedback on whole task performance.…”

“…In clinical practice, health professionals perform psychomotor tasks (also termed procedural, clinical or technical skills) that are unique to their discipline. Most clinical skills, when performed on a conscious patient require simultaneous communication with the patient, using a set of vocabulary distinct to the discipline Kneebone et al 2002;Pugh et al 2015). Historically, priority has been given to learning the procedural task, i.e.…”

Using the Entrustable Professional Activities Framework in the Assessment of Procedural Skills

“…[10][11][12][13] These skills include the ability to communicate effectively (eg, obtaining informed consent); to collaborate with other professionals; to act in a professional manner; and to exercise clinical reasoning and judgment. 14 There have been attempts to assess procedural skills in context, [15][16][17] but it remains unclear which nontechnical skills are most important to assess.…”

Expert Panel Consensus on Assessment Checklists for a Rheumatology Objective Structured Clinical Examination

“…Additionally, OSCEs show promise for use to assess learners' progress at different training levels, a feature that may lead to increased use of OSCEs (2). Through contrived, repeatable scenarios, an OSCE allows standardization of assessment such that performance across learners can be compared, which is not possible when observing actual patient-based clinical interactions.…”

“…Through contrived, repeatable scenarios, an OSCE allows standardization of assessment such that performance across learners can be compared, which is not possible when observing actual patient-based clinical interactions. Additionally, OSCEs show promise for use to assess learners' progress at different training levels, a feature that may lead to increased use of OSCEs (2).…”