Sounds of COVID-19: exploring realistic performance of audio-based digital testing

Abstract

To identify Coronavirus disease (COVID-19) cases efficiently, affordably, and at scale, recent work has shown how audio (including cough, breathing and voice) based approaches can be used for testing. However, there is a lack of exploration of how biases and methodological decisions impact these tools’ performance in practice. In this paper, we explore the realistic performance of audio-based digital testing of COVID-19. To investigate this, we collected a large crowdsourced respiratory audio dataset through a mobile app, alongside symptoms and COVID-19 test results. Within the collected dataset, we selected 5240 samples from 2478 English-speaking participants and split them into participant-independent sets for model development and validation. In addition to controlling the language, we also balanced demographics for model training to avoid potential acoustic bias. We used these audio samples to construct an audio-based COVID-19 prediction model. The unbiased model took features extracted from breathing, coughs and voice signals as predictors and yielded an AUC-ROC of 0.71 (95% CI: 0.65–0.77). We further explored several scenarios with different types of unbalanced data distributions to demonstrate how biases and participant splits affect the performance. With these different, but less appropriate, evaluation strategies, the performance could be overestimated, reaching an AUC up to 0.90 (95% CI: 0.85–0.95) in some circumstances. We found that an unrealistic experimental setting can result in misleading, sometimes over-optimistic, performance. Instead, we reported complete and reliable results on crowd-sourced data, which would allow medical professionals and policy makers to accurately assess the value of this technology and facilitate its deployment.

Diagnosing COVID-19 using artificial intelligence: a comprehensive review

“…The model performed exceptionally well by employing the auto-regressive predictive coding technique (Harvill et al 2021). Han et al (2022) proposed a study that scrutinized the realistic performance of acoustics in the diagnosis of coronavirus. The dataset consisted of 5240 audio samples from 2478 participants.…”

Challenges and Opportunities of Deep Learning for Cough-Based COVID-19 Diagnosis: A Scoping Review

“…Moreover, when the performance of one neural network model is assessed for several separate subgroups, differences arise. Han et al [ 34 ] thoroughly detailed the performance of a CNN for distinct subgroups of gender, age, symptom manifestation, medical history, and smoking status. When separated, the subgroups yielded higher performance (the difference was approximately 10% on average).…”

Mobile Sensing in the COVID-19 Era: A Review

“…(1) Design. Remote detection (see in Table 4) majorly leverages (1) audio data, e.g., respiratory sounds collected from microphones equipped in smartphones (N = 8 of 15) [17][18][19][20][21][22][23][24], and (2) physiological data, e.g., blood oxygen sensed by smartwatches (N = 6 of 15) [25][26][27][28][29][30], to identify whether a user has been infected through machine learning-based approaches. Some of them [20,28,30,31] also leverage/ incorporate self-reported symptom data to improve the effectiveness of detection.…”

“…The prerequisite of owning wearable devices and the ability to self-report COVID-19-related symptoms might raise the economic and education bars of technology adoption for remote detection [32], resulting in biases in population coverage. Furthermore, all these methods rely on large-scale data collection for training datasets, while unbalanced data collections might cause also biases in prediction results under varying demographics, languages, devices, and physiological/respiratory conditions [22,33].…”