The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia

Abstract

Objectives To investigate the quality of diagnostic and triage advice provided by free website and mobile application symptom checkers (SCs) accessible in Australia. Design 36 SCs providing medical diagnosis or triage advice were tested with 48 medical condition vignettes (1170 diagnosis vignette tests, 688 triage vignette tests). Main outcome measures Correct diagnosis advice (provided in first, the top three or top ten diagnosis results); correct triage advice (appropriate triage category recommended). Results The 27 diagnostic SCs listed the correct diagnosis first in 421 of 1170 SC vignette tests (36%; 95% CI, 31–42%), among the top three results in 606 tests (52%; 95% CI, 47–59%), and among the top ten results in 681 tests (58%; 95% CI, 53–65%). SCs using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests (95% CI, 40–57%), compared with 32% (95% CI, 26–38%) for other SCs. The mean rate of first correct results for individual SCs ranged between 12% and 61%. The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%; 95% CI, 44–54%). Appropriate triage advice was more frequent for emergency care (63%; 95% CI, 52–71%) and urgent care vignette tests (56%; 95% CI, 52–75%) than for non‐urgent care (30%; 95% CI, 11–39%) and self‐care tests (40%; 95% CI, 26–49%). Conclusion The quality of diagnostic advice varied between SCs, and triage advice was generally risk‐averse, often recommending more urgent care than appropriate.

Providing pediatric well-care and sick visits in the COVID-19 pandemic era: the recommendations of the Italian pediatric society

“…A recent survey found out that more than half of the respondents (54%) used the search engine at least weekly to look up medical questions and symptoms and more than 40% use Google as the only source of information on health. The results may have important repercussions on people' health, given that most of the time, "Dr Google" doesn't provide the right diagnosis [13]. Pediatricians should continue providing routine preventive and other nonemergency care, discussing with caregivers on the benefit of attending a in-person visit, immunizations and screenings to avoid both missing visits and wrong diagnosis trusting on-line information.…”

Will online symptom checkers improve health care in Australia?

“…Another important finding by Hill and her co‐authors is that symptom checkers are generally risk‐averse, recommending more urgent care than needed, echoing the findings of overseas analyses . Nevertheless, safety problems that can arise when less urgent care is recommended for serious conditions should not be dismissed, even if such cases are rare …”

“…Symptom checkers claiming to use artificial intelligence (AI) methods outperformed other checkers in the vignette‐based analysis . But characteristics other than computational sophistication also influence the safety and utility of symptom checkers, including updatability, localisation, failing gracefully, and intent.…”

“…In a public health crisis, decisions about where and how to seek care can change daily. When looking online for symptom checkers to include in their study, Hill and colleagues searched, among other things, for flu‐like symptoms (“cough, fever, headache”). It is safe to assume that none of the included symptom checkers would have been equipped to keep up with the emerging COVID‐19 pandemic.…”

“…In this issue of the MJA , Hill and colleagues used a vignette‐based approach to evaluate the ability of 36 symptom checkers available in Australia to correctly diagnose and triage people. The vignettes included scenarios that health providers encounter in Australia, including shingles, heart attacks, and viral upper respiratory infections.…”