Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation

Abstract

Background Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment. Objective This study aims to revisit the landmark index study to investigate whether and how symptom checkers’ capabilities have evolved since 2015 and how they currently compare with laypersons’ stand-alone triage appraisal. Methods In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons’ triage capability. Results We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions. Conclusions Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Determinants of Laypersons’ Trust in Medical Decision Aids: Randomized Controlled Trial

“…In every condition, they had to appraise one and the same case vignette by deciding whether the fictitious patient required health care or self-care was sufficient. Although there are other urgency levels in symptom checkers, we chose this binary decision as the question of whether to seek care at all is the first decision patients must make [ 15 , 20 ]. The participants were tasked to appraise the case vignette twice: once before receiving advice from the decision aid (initial stand-alone assessment) and once after receiving the advice.…”

“…A vignette-based experimental study found that on average, symptom checkers currently do not outperform laypersons in terms of disposition accuracy. However, best-in-class apps seem superior to laypersons [ 19 , 20 ].…”

Triage Errors in Primary and Pre–Primary Care

“…The accuracy of digital symptom checkers in providing triage has been met with skepticism. There is limited evidence in this area, but vignette studies have suggested that triage error rates have been shown to be high for digital symptom checkers [ 23 , 24 ]. One study compared 12 publicly available symptom checkers and reported that only 51% of triage decisions for the top 5 diagnoses were correct [ 23 ].…”

“…Triage advice provided by symptom checkers is found to be more risk averse than that provided by health care professionals [ 30 , 32 ], with 85% of the users advised to see their doctor in one study [ 33 ]. However, in a 5-year follow-up evaluation study, it was observed that symptom checkers in 2020 are less risk averse (odds of 1.11:1, overtriage errors to undertriage errors) than in 2015 (odds of 2.82:1) [ 24 ]. Triage errors in emergencies, nevertheless, are still high, with 40% of emergency cases being missed by symptom checkers [ 24 ].…”

“…However, in a 5-year follow-up evaluation study, it was observed that symptom checkers in 2020 are less risk averse (odds of 1.11:1, overtriage errors to undertriage errors) than in 2015 (odds of 2.82:1) [ 24 ]. Triage errors in emergencies, nevertheless, are still high, with 40% of emergency cases being missed by symptom checkers [ 24 ].…”