Use of an error‐focused checklist to identify incompetence in lumbar puncture performances

Abstract

ContextChecklists are commonly used in the assessment of procedural competence. However, on most checklists, high scores are often unable to rule out incompetence as the commission of a few serious procedural errors typically results in only a minimal reduction in performance score. We hypothesised that checklists constructed based on procedural errors may be better at identifying incompetence.ObjectivesThis study sought to compare the efficacy of an error-focused checklist and a conventionally constructed checklist in identifying procedural incompetence.MethodsWe constructed a 15-item error-focused checklist for lumbar puncture (LP) based on input from 13 experts in four Canadian academic centres, using a modified Delphi approach, over three rounds of survey. Ratings of 18 video-recorded performances of LP on simulators using the error-focused tool were compared with ratings obtained using a published conventional 21-item checklist. Competence/incompetence decisions were based on global assessment. Diagnostic accuracy was estimated using the area under the curve (AUC) in receiver operating characteristic analyses.ResultsThe accuracy of the conventional checklist in identifying incompetence was low (AUC 0.11, 95% confidence interval [CI] 0.00–0.28) in comparison with that of the error-focused checklist (AUC 0.85, 95% CI 0.67–1.00). The internal consistency of the error-focused checklist was lower than that of the conventional checklist (α = 0.35 and α = 0.79, respectively). The inter-rater reliability of both tools was high (conventional checklist: intraclass correlation coefficient [ICC] 0.99, 95% CI 0.98–1.00; error-focused checklist: ICC 0.92, 95% CI 0.68–0.98).ConclusionsDespite higher internal consistency and inter-rater reliability, the conventional checklist was less accurate at identifying procedural incompetence. For assessments in which it is important to identify procedural incompetence, we recommend the use of an error-focused checklist.

Construct validity of a novel assessment tool for ultrasound‐guided axillary brachial plexus block

“…A previous study concluded that error‐focused checklists are superior to conventional checklists for determining procedural incompetence . In this study, two types of error metrics, procedural and critical errors were defined.…”

Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills

“…Of the 31 items on the tool, only 16 (52%) were felt to be critically important. Although critical items on clinical and procedural skills have previously been published, 30,[48][49][50][51] to our knowledge, they have not been established for general POCUS skills. Delineating what items are critical is important for POCUS for 2 reasons.…”

“…26,27 However, checklists risk ''rewarding thoroughness,'' allowing the successful completion of multiple trivial items while masking the commission of a single serious error. [27][28][29][30][31] As such, there is a need to establish which checklist items are critical in POCUS, such that incompetent performances are appropriately identified.…”

Paediatric lumbar punctures: How do paediatric and emergency doctors differ?

“…Improved teaching and learning appears to require simulated practice with part‐task trainers, although this does not necessarily ensure procedural competence . Error‐based checklists have been suggested as a means to assess procedural competence in the simulated setting …”

“…9,10 Error-based checklists have been suggested as a means to assess procedural competence in the simulated setting. 11 From the data gathered, paediatricians and paediatric trainees had a greater tendency to perform fundoscopy in infants compared to their emergency counterparts. Brain shifts and ischaemia are the two major consequences of an increase in ICP.…”