Diagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study

Abstract

Background A large number of web-based COVID-19 symptom checkers and chatbots have been developed; however, anecdotal evidence suggests that their conclusions are highly variable. To our knowledge, no study has evaluated the accuracy of COVID-19 symptom checkers in a statistically rigorous manner. Objective The aim of this study is to evaluate and compare the diagnostic accuracies of web-based COVID-19 symptom checkers. Methods We identified 10 web-based COVID-19 symptom checkers, all of which were included in the study. We evaluated the COVID-19 symptom checkers by assessing 50 COVID-19 case reports alongside 410 non–COVID-19 control cases. A bootstrapping method was used to counter the unbalanced sample sizes and obtain confidence intervals (CIs). Results are reported as sensitivity, specificity, F1 score, and Matthews correlation coefficient (MCC). Results The classification task between COVID-19–positive and COVID-19–negative for “high risk” cases among the 460 test cases yielded (sorted by F1 score): Symptoma (F1=0.92, MCC=0.85), Infermedica (F1=0.80, MCC=0.61), US Centers for Disease Control and Prevention (CDC) (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Cleveland Clinic (F1=0.40, MCC=0.07), Providence (F1=0.40, MCC=0.05), Apple (F1=0.29, MCC=-0.10), Docyet (F1=0.27, MCC=0.29), Ada (F1=0.24, MCC=0.27) and Your.MD (F1=0.24, MCC=0.27). For “high risk” and “medium risk” combined the performance was: Symptoma (F1=0.91, MCC=0.83) Infermedica (F1=0.80, MCC=0.61), Cleveland Clinic (F1=0.76, MCC=0.47), Providence (F1=0.75, MCC=0.45), Your.MD (F1=0.72, MCC=0.33), CDC (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Apple (F1=0.70, MCC=0.25), Ada (F1=0.42, MCC=0.03), and Docyet (F1=0.27, MCC=0.29). Conclusions We found that the number of correctly assessed COVID-19 and control cases varies considerably between symptom checkers, with different symptom checkers showing different strengths with respect to sensitivity and specificity. A good balance between sensitivity and specificity was only achieved by two symptom checkers.

A Novel Olfactory Self-Test Effectively Screens for COVID-19

“…Does this minimal difference (Figure 4A) imply that single-odorant olfactory testing has no advantage over symptom checking? Although some symptom checkers have reported even stronger results than those we obtain here 19 , there are two critical points where symptom-checking alone largely fails. One such point is with individuals who all have somatic symptoms such as fever, etc., but don’t have COVID-19.…”

“…This need has been partially met with online symptom checkers. These have obtained some very impressive results 3,19 , yet as previously noted, they inherently all-out fail at two critical points: One is in individuals who are ill, but not with COVID-19. If an individual has a cough, fever, and a headache, symptom-checkers will most likely estimate them to have COVID-19.…”

Symptoms associated with a COVID-19 infection in the general population of Vienna

“…The Vienna Social Fund (FSW), the Public Health Services of the City of Vienna (MA15) and the company Symptoma mutually developed the chatbot based on the database, artificial intelligence, and algorithms of Symptoma [3], [6], [7]. During the chatbot conversation, the user is answering a series of 13 yes/no questions about symptoms.…”

Machine Learning in Healthcare Communication

“…The result shows different symptoms checkers showing different strengths in terms of sensitivity and specificity. Only two checkers had a harmonic balance between specificity and sensitivity [93]. With the advancement in NLP, the chatbot can answer the user's questions automatically.…”