Appraisal of a scoring instrument for training and testing neonatal intubation skills

Abstract

ObjectiveTo determine the validity, reliability, feasibility and applicability of a neonatal intubation scoring instrument.DesignProspective observational study.SettingSimulation-based research and training centre (Center for Advanced Pediatric and Perinatal Education), California, USA.SubjectsForty clinicians qualified for neonatal intubation.InterventionsVideotaped elective intubations on a neonatal patient simulator were scored by two independent raters. One rater scored the intubations twice. We scored the preparation of equipment and premedication, intubation performance, tube position/fixation, communication, number of attempts, duration and successfulness of the procedure.Main outcome measuresIntraclass correlation coefficients (ICC) were calculated for intrarater and inter-rater reliability. Kappa coefficients for individual items and mean kappa coefficients for all items combined were calculated. Construct validity was assessed with one-way analysis of variance using the hypothesis that experienced clinicians score higher than less experienced clinicians. The approximate time to score one intubation and the instrument’s applicability in another setting were evaluated.ResultsICCs for intrarater and inter-rater reliability were 0.99 (95% CI 0.98 to 0.99) and 0.89 (95% CI 0.35 to 0.96), and mean kappa coefficients were 0.93 (95% CI 0.85 to 1.01) and 0.71 (95% CI 0.56 to 0.92), respectively. There were no differences between the more and less experienced clinicians regarding preparation, performance, communication and total scores. The experienced group scored higher only on tube position/fixation (p=0.02). Scoring one intubation took approximately 15 min. Our instrument, developed in The Netherlands, could be readily applied in the USA.ConclusionsOur scoring instrument for simulated neonatal intubations appears to be reliable, feasible and applicable in another centre. Construct validity could not be established.

Validation of an Instrument for Real-Time Assessment of Neonatal Intubation Skills: A Randomized Controlled Simulation Study

“…Based on this study and its prequel, 19 it can be concluded that our NISI is valid, reliable, and practical. It can be used to assess (simulated) neonatal intubations in real time and remotely on video.…”

“…In the present study, we evaluated the construct validity and reliability of real-time assessment of our previously published NISI. 19 Using an extreme groups design, we found a significant positive correlation between intubation experience and performance. In addition, we demonstrated that structured, individualized feedback improved subsequent neonatal intubation performance.…”

“…22 The data from our prequel study were also insufficiently appropriate to this end, since those data were collected with a significantly different study population. 19 Second, a sample size estimation requires a statement on what would constitute a clinically meaningful difference. 22 Paradoxically, we had to validate our assessment instrument first to be able to conduct follow-up studies to assess what differences in score, as identified by the assessment instrument, would actually translate into clinically important differences in intubation performance.…”

“…In 2016, a study performed by our research group established the reliability, feasibility, and applicability of a scoring instrument for video-based assessment of neonatal intubation in a simulated setting. 19 However, this study could not establish the instrument's construct validity, possibly because the study groups were insufficiently divergent in terms of neonatal intubation experience. It also remained to be determined if the instrument could be reliably used for real-time assessment in addition to remote video-based assessment.…”

“…For a detailed description of the instrument's development, we refer to Bouwmeester et al 19 All 28 items of our NISI were classified under the subheadings Preparation, Performance, Post-intubation care, and Communication (►Supplementary Material 1, available in the online version). Most items were scored trichotomously: 0 points (not done), 1 point (partially or not timely done), and 2 points (completely and timely done).…”