Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Abstract

Background Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. Objective The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). Methods We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. Results A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low. Conclusions This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Evaluating self-triage accuracy of laypeople, symptom-assessment apps, and large language models: A framework for case vignette development using a representative design approach (RepVig)

“…In this sense, also traditional vignettes can provide valuable insights, for instance, to examine how SAAs might perform when being used by clinicians . Similarly, studying ED cases (as done by Fraser et al 49 ) could help determine if patients visiting the ED might have opted for primary care had they consulted an SAA beforehand. With a sample of primary care cases, the potential effect of SAA on the redistribution of care seeking efforts could be assessed from a systems perspective.…”

Accuracy of Dental Symptom Checker Web Application in the Singapore Military Population

“…The user sentiments for this online SC were generally positive, as compared to the use of Ada Health App in the ED which reported a technology acceptance usability of mean score 84.6% (40). When implemented on a larger scale, some studies opined that digital tools like SCs might promote self-care and may positively impact the delivery of healthcare and reduce the burden on emergency medical services (41,42).…”

Diagnostic Performance Comparison between Generative AI and Physicians: A Systematic Review and Meta-Analysis

“…After screening, 54 studies were included in the metaanalysis of generative AI diagnostic performance 12,13, and 13 studies in the comparative analysis between generative AI models and physicians (Figure 1 and Table 1). 32,33,[35][36][37][38][39][40][41]49,52,58,62 The most evaluated models were GPT-4 4 (31 articles) and GPT-3.5 (28), while models such as GPT-4V ( 6), PaLM2 8 (3), Llama 2 6 (2), Prometheus (2), GPT-3 3 (1), Glass ( 1 were represented, as well as Gastroenterology, Cardiology, Pediatrics, Otolaryngology, Urology, Endocrinology, Gynecology, Orthopedic surgery, Rheumatology, Psychiatry, and Plastic surgery with one article each. Regarding model tasks, free text tasks were the most common, with 47 articles, followed by choice tasks at 13.…”

“…Thirteen studies compared the performance of generative AI models with physicians. 32,33,[35][36][37][38][39][40][41]49,52,58,62 GPT-4 (8 articles) was the most frequently evaluated, followed by GPT-3.5 (7), GPT-4V (2), Llama 2 (1), and GPT-3 (1).…”