Accuracy of online symptom checkers and the potential impact on service utilisation

Abstract

<jats:p>Objectives: The aims of this study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems. Setting: Publicly available symptom checkers Participants: Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes was developed and systematically run through each system by a non-clinical researcher. Primary and secondary outcome measures: System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette. Results: Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being listed first on 37.7% (Range 22.2 to 72.0%) of occasions and present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems. Conclusions: There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.</jats:p>

Quality of condition suggestions and urgency advice provided by the Ada symptom assessment app evaluated with vignettes optimised for Australia

“…The finding of this study, namely that the Ada app has relatively high accuracy with regard to condition suggestion and urgency compared with other available SAAs, reflects the findings of other studies (Nateqi et al 2019;Ceney et al 2020;Gilbert et al 2020). The Ada app was recently compared with general practitioners (GPs) and competitor apps in a 200-vignette study (Gilbert et al 2020).…”

Avey: An Accurate AI Algorithm for Self-Diagnosis

“…Building on prior related work [12, 22, 27, 36, 52, 53], we adopted a clinical vignette approach to measure the performance of Avey alongside several other checkers. A seminal work at Harvard Medical School has established the value of this approach [22, 52, 53] for testing checkers, especially that it has been also a common method to test physicians on their diagnosis abilities [53].…”

“…They also observed that AI-based checkers outperform rule-based ones (i.e., checkers that do not use AI). Akin to Hill et al [27], Ceney et al [12] detected a significant variation in accuracy across 12 checkers, ranging from 22.2% (CAIDR [11]) to 72% (Ada) using 𝑀5.…”

“…Checkers are inherently scalable (i.e., they can assess millions of people instantly and concurrently) and universally available. Besides, they promise to provide patients with necessary high-quality, evidence-based information [55], reduce unnecessary medical visits [1, 10, 42, 45], alleviate the pressure on healthcare systems [3], improve accessibility to timely diagnosis [1], and guide patients to the most appropriate care pathways [12], to mention just a few.…”

“…Checkers are increasingly becoming an integral part of digital health, with more than 15 million users per month [52] that are likely to keep growing [12]. A UK-based study that engaged 1,071 patients found that more than 70% of individuals between the ages of 18 and 39 years would use a checker [16].…”