Background Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. Objective The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). Methods We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. Results A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low. Conclusions This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

No abstract

BACKGROUND Symptom checkers (SCs) are diagnostic decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnosis and triage advice and assist users in seeking the appropriate level of care. Little evidence is available on their diagnostic and triage accuracy with direct use by patients for urgent conditions. OBJECTIVE To determine the diagnostic and triage accuracy and usability of a symptom checker in use by for patients presenting to an emergency department METHODS We recruited a convenience sample of English-speaking patients presenting for care in an urban emergency department. Each consenting patient used a leading SC from Ada Health prior to ED evaluation. Diagnostic accuracy was evaluated, comparing (1) the SC’s diagnoses and (2) those of 3 independent emergency physicians viewing the patient-entered symptom data, to (3) the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. Patients completed a usability survey based on the Technology Acceptance Model. RESULTS Forty participants (80% of those approached, mean age 40.1 years (range 18 – 76); 65% female, 68% White, 48% Hispanic or Latino, completed the SC assessment and usability survey. Sensitivity for at least one of the ED final diagnoses by Ada (based on it’s top 5 diagnoses) was 70.0% (95% CI 54% – 86%) close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated Ada triage decisions as 62% fully agree, and 24% safe but too cautious. It was rated unsafe and too risky in 22% by at least one physician, 14% by two and 5% by all 3, although none of the under-triaged patients suffered an adverse event in the ED. Some cases only had symptoms recorded for their ED diagnosis, limiting analysis. Usability was rated highly, participants agreed or strongly agreed a mean of 84.6% with the 7 TAM usability questions, although “satisfaction” and “enjoyment” were rated low. CONCLUSIONS This study provides preliminary evidence that a SC can provide acceptable usability and diagnostic accuracy for patients with a variety of urgent conditions. 14% of SC triage recommendations were deemed unsafe and too risky by at least 2 physicians based on the symptoms recorded. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.