Validity of a deep learning algorithm for detecting wheezes and crackles from lung sound recordings in adults

Abstract

We aimed at evaluating deep learning algorithms for detecting wheezes and crackles developed based on sound files from 4033 adults in two samples of sound files not used in the algorithm development. In sample A, ground truth was established by experienced raters in 615 files from the Tromsø population study. Sample B contained 120 sound files from a previous interobserver study with ground truth determined by four experts. The algorithms' probability scores for wheezes and crackles were evaluated against the ground truth by calculating Area Under Curve (AUC). Agreements between the algorithm and the human annotations were also assessed by Kappa statistics. In sample A the AUC was 0.88 (95% CI 0.84 - 0.92) for wheezes and 0.88 (95% CI 0.84 - 0.92) for crackles. The kappa agreement between dichotomized labelling and the ground truth was 0.63 (95% CI 0.56 - 0.71) for wheezes and 0.68 (95% CI 0.60 - 0.75) for crackles. The corresponding kappa agreements between the human raters were 0.47. In sample B, an AUC of 0.99 (95% CI 0.98 - 1.0) was reached for wheezes and 0.95 (95% CI 0.89 - 1.0) for crackles with corresponding kappas of 0.78 (95% CI 0.58 - 0.99) and 0.75 (95% CI 0.59 - 0.92). The corresponding mean kappas between the ground truth and 24 observers were 0.68 and 0.55. The algorithm agreed substantially with ground truth, and with higher kappa agreements than observed between human annotators.