Does training improve diagnostic accuracy and inter-rater agreement in applying the Berlin radiographic definition of acute respiratory distress syndrome? A multicenter prospective study

Abstract

BackgroundPoor inter-rater reliability in chest radiograph interpretation has been reported in the context of acute respiratory distress syndrome (ARDS), although not for the Berlin definition of ARDS. We sought to examine the effect of training material on the accuracy and consistency of intensivists’ chest radiograph interpretations for ARDS diagnosis.MethodsWe conducted a rater agreement study in which 286 intensivists (residents 41.3%, junior attending physicians 35.3%, and senior attending physician 23.4%) independently reviewed the same 12 chest radiographs developed by the ARDS Definition Task Force (“the panel”) before and after training. Radiographic diagnoses by the panel were classified into the consistent (n = 4), equivocal (n = 4), and inconsistent (n = 4) categories and were used as a reference. The 1.5-hour training course attended by all 286 intensivists included introduction of the diagnostic rationale, and a subsequent in-depth discussion to reach consensus for all 12 radiographs.ResultsOverall diagnostic accuracy, which was defined as the percentage of chest radiographs that were interpreted correctly, improved but remained poor after training (42.0 ± 14.8% before training vs. 55.3 ± 23.4% after training, p < 0.001). Diagnostic sensitivity and specificity improved after training for all diagnostic categories (p < 0.001), with the exception of specificity for the equivocal category (p = 0.883). Diagnostic accuracy was higher for the consistent category than for the inconsistent and equivocal categories (p < 0.001). Comparisons of pre-training and post-training results revealed that inter-rater agreement was poor and did not improve after training, as assessed by overall agreement (0.450 ± 0.406 vs. 0.461 ± 0.575, p = 0.792), Fleiss’s kappa (0.133 ± 0.575 vs. 0.178 ± 0.710, p = 0.405), and intraclass correlation coefficient (ICC; 0.219 vs. 0.276, p = 0.470).ConclusionsThe radiographic diagnostic accuracy and inter-rater agreement were poor when the Berlin radiographic definition was used, and were not significantly improved by the training set of chest radiographs developed by the ARDS Definition Task Force.Trial registrationThe study was registered at ClinicalTrials.gov (registration number NCT01704066) on 6 October 2012.Electronic supplementary materialThe online version of this article (doi:10.1186/s13054-017-1606-4) contains supplementary material, which is available to authorized users.

Extravascular lung water measurements in acute respiratory distress syndrome: why, how, and when?

“…The existence and severity of pulmonary edema are generally evaluated based on physical examination, patient history, laboratory examination, and chest radiographic findings [ 2 ]. However, the interpretation of these parameters, including chest X ray, is often affected by subjective factors that may cause interobserver error, even among experts [ 2 , 7 , 8 ▪▪ , 9 ▪▪ ]. Objective diagnosis of pulmonary edema can be made if EVLW is evaluated quantitatively at the bedside using the transpulmonary thermodilution (TPTD) technique [ 1 , 10 ] (Fig.…”

“…This conclusion was supported by a recent multicenter prospective study of interrater agreement, in which 286 intensivists independently reviewed the same 12 chest radiographs developed by the panels, before and after training. Radiographic diagnostic accuracy and interrater agreement were found to be poor when the Berlin radiographic definition was used and were not significantly improved by the training set of chest radiographs developed by the Task Force [ 8 ▪▪ ].…”

ERS statement on chest imaging in acute respiratory failure

“…A recent study evaluated whether diagnostic accuracy could be improved by an educational intervention based on a set of chest radiographs. Though some improvement in diagnostic accuracy after the educational intervention was observed, the overall accuracy remained poor [5]. A multicentre randomised trial using an online educational module of the Berlin definition on ARDS failed to demonstrate any improvement in the accuracy or in the identification of ARDS patients among critical care clinicians and research staff [6].…”

Lung imaging: how to get better look inside the lung

“…Nonetheless, if used as a monitoring tool frequent exposure to ionizing radiations can be a concern, without guaranteeing high sensitivity and specificity, especially in the earlier stages (11).…”