Validation of an instrument for the evaluation of exchange transfusion (INEXTUS) via an OSCE

Abstract

Introduction Exchange transfusion is the treatment of choice for patients with severe hyperbilirubinemia who do not respond to phototherapy. This procedure is highly complex and requires substantial expertise to perform, however it´s not done frequently enough to guarantee adequate training. Traditional learning scenarios do not have a space reserved for teaching this procedure or an instrument that fully and objectively evaluates the skills that a professional must acquire. Objectives The purpose was to construct and evaluate the INEXTUS instrument´s validity evidence relevant to internal structure, in a simulated scenario through the performance of an objective structured clinical exam (OSCE). Materials and methods The Delphi consensus methodology was utilized to design the instrument; six experts participated through three rounds using the Google Forms platform. The categories and items previously obtained were subjected to validation by nine experts through a dichotomous survey. Prior to data collection, the evaluators were trained through a pilot test with 10 medical students. Subsequently, all residents of a paediatric programme were evaluated through the OSCE methodology in a simulated scenario, with 6 stations, of a clinical case of a new-born with an explicit need for exchange transfusion. During their participation in the scenario, the residents were first evaluated with the instrument developed. Additionally, audio and video filming of all students who participated was performed with the aim of conducting a second evaluation two weeks after the first four evaluators participated. Results The final INEXTUS instrument consists of 46 subitems grouped into 23 items divided into 6 categories, demonstrating an inter-rater intraclass correlation coefficient of 0.96 (95% CI 0.94, 0.98 p-value < 0.001). For the Fleiss Kappa of the 23 items evaluated, concordance was evaluated for 14 items but could not be determined for the 9 remaining items because all the ratings were equal, either because the items were not performed or they were all performed adequately. Of the 14 items, 9 good scores were obtained (95% CI 0.61 to 0.8; p value < 0.001), and 5 very good scores were obtained (95% CI 0.81 to 1; p value < 0.001). Conclusions The INEXTUS instrument evaluates exchange transfusion skills in medical personnel in training in simulated scenarios using the OSCE methodology; it has high validity and reliability and is a high-impact educational tool.

eOSCE stations live versus remote evaluation and scores variability

“…More experienced raters scored higher than juniors. Regarding students’ ICC, they are slightly lower than those reported in previous publications on interrater-reliability in on-site OSCEs [ 20 , 21 ]. Per instance, in this study by Hurley et al which objective was to assess inter-observer reliability and observer accuracy as a function of OSCE checklist length.…”