Exploring the use of videotaped objective structured clinical examination in the assessment of joint examination skills of medical students

Abstract

Objective. Objective structured clinical examination (OSCE) is a key part of medical student assessment. Currently, assessment is performed by medical examiners in situ. Our objective was to determine whether assessment by videotaped OSCE is as reliable as live OSCE assessment. Methods. Participants were 95 undergraduate medical students attending their musculoskeletal week at Freeman Hospital, Newcastle (UK). Student performance on OSCE stations for shoulder or knee examinations was assessed by experienced rheumatologists. The stations were also videotaped and scored by a rheumatologist independently. The examinations consisted of a 14-item checklist and a global rating scale (GRS). Results. Mean values for the shoulder OSCE checklist were 17.9 by live assessment and 17.4 by video (n ‫؍‬ 50), and 20.9 and 20.0 for live and video knee assessment, respectively (n ‫؍‬ 45). Intraclass correlation coefficients for shoulder and knee checklists were 0.55 and 0.58, respectively, indicating moderate reliability between live and video scores for the OSCE checklists. GRS scores were less reliable than checklist scores. There was 84% agreement in the classification of examination grades between live and video checklist scores for the shoulder and 87% agreement for the knee ( ‫؍‬ 0.43 and 0.51, respectively; P < 0.001). Conclusion. Video OSCE has the potential to be reliable and offers some advantages over live OSCE including more efficient use of examiners' time, increased fairness, and better monitoring of standards across various schools/sites. However, further work is needed to support our findings and to implement and evaluate the quality assurance issues identified in this work before justifiable recommendations can be made.

Direct Observation versus Endoscopic Video Recording-Based Rating with the Objective Structured Assessment of Technical Skills for Training of Laparoscopic Cholecystectomy

“…The calculated ICC values considering all raters indicated that there were variations in scoring. It should be noted that although we consider our ICCs as moderate, discrepancies exist across studies for ICC interpretation [5,13,25,26,27,28]. Consistent with our findings, the literature has shown interrater reliability between video and direct scoring to be moderate or higher [3,13,18].…”

“…We surmise that an intensified mandatory rater training would have increased interrater consistency in scoring and thus improved the ICCs. Data from Vivekananda et al [13] suggest that video scores are reproducible and provide high intrarater reliability. We provide no evidence to support or oppose this, but find it likely to be true, considering the fact that we found direct OSATS and video OSATS ratings to have high internal consistency between GRS and STS scores, as calculated by Cronbach's α.…”

“…This can be explained by benevolence, as well as the removal of bias in video recordings and the inability to see the whole surgical environment on endoscopic videos. Live scoring offers the advantage of allowing raters the ability to clarify any uncertainty in a trainee's knowledge, but remains simultaneously disadvantageous in that rater-trainee interactions may introduce bias [13]. Direct and video scores correlated significantly for OSATS, GRS, and STS, implying that video rating may be used as an alternative to direct rating.…”

Videotaped OSPE: Is This a Right Procedure to Assess Health Science Students’ Performance? — A Pilot Study

“…Video shooting and learning in the modern education ease the student's attention and interactions. Videotaped station can be a standard practical assessment procedure in marking, even with off campus examiners, and is also useful to retrieve the evidence of students' performance in future [9]. Experts had mentioned in early studies to modify the OSPE procedures based on local circumstances in educational institutes [8].…”

Scoring Objective Structured Clinical Examinations Using Video Monitors or Video Recordings

“…5 Vivekananda-Schmidt and colleagues investigated inter-rater reliability between real-time and video-recorded OSCEs of 95 third-year medical students' shoulder and knee examinations using the intraclass correlation coefficient (ICC). 5 Real-time OSCE encounters were scored by physicians training in rheumatology who were present in the room with the student and standardized patient during the encounter. Later, a consultant rheumatologist observed a video recording of the encounter and independently scored each student's performance.…”