What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study

Abstract

ObjectiveAssess the suitability of clinical vignettes in benchmarking the performance of online symptom checkers (OSCs).DesignObservational study using a publicly available free OSC.ParticipantsHealthily OSC, which provided consultations in English, was used to record consultation outcomes from two lay and four expert inputters using 139 standardised patient vignettes. Each vignette included three diagnostic solutions and a triage recommendation in one of three categories of triage urgency. A panel of three independent general practitioners interpreted the vignettes to arrive at an alternative set of diagnostic and triage solutions. Both sets of diagnostic and triage solutions were consolidated to arrive at a final consolidated version for benchmarking.Main outcome measuresSix inputters simulated 834 standardised patient evaluations using Healthily OSC and recorded outputs (triage solution, signposting, and whether the correct diagnostic solution appeared first or within the first three differentials). We estimated Cohen’s kappa to assess how interpretations by different inputters could lead to divergent OSC output even when using the same vignette or when compared with a separate panel of physicians.ResultsThere was moderate agreement on triage recommendation (kappa=0.48), and substantial agreement on consultation outcomes between all inputters (kappa=0.73). OSC performance improved significantly from baseline when compared against the final consolidated diagnostic and triage solution (p<0.001).ConclusionsClinical vignettes are inherently limited in their utility to benchmark the diagnostic accuracy or triage safety of OSC. Real-world evidence studies involving real patients are recommended to benchmark the performance of OSC against a panel of physicians.

The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review

“…First, five of the included studies used simulated patient vignettes, which are unlikely to represent the complexity of real patients 25 – 27 . Future work could include real-life patient data to increase software exposure to the nuances of real-life patient populations.…”

“…While vignettes or synthetic data with typical disease patterns and standardised presentations may be viewed as a viable and accessible means of training a symptom checker, this may not negate the need for testing using real-world cases 27 . Clinical vignettes are inherently limited in their utility to assess the diagnostic accuracy or triage safety of symptom checkers.…”

“…Clinical vignettes are inherently limited in their utility to assess the diagnostic accuracy or triage safety of symptom checkers. Real-world evidence studies involving real patients have been recommended to examine the performance of symptom checkers against clinicians 27 . The importance of using real-world data within digital health technology development is also strongly supported by regulatory authorities, as evidenced by the introduction of the ‘Real World Evidence’ program by the FDA, which have also approved databases with real-world patient data to support this initiative 31 – 33 .…”

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

“…OSCs are most often validated using a set of clinical case 'vignettes' [4][5][6]11,12]. Each vignette represents a possible clinical scenario or 'case' and contains information such as key patient demographics, relevant medical history, and symptoms.…”

“…The lack of consensus on OSC clinical evaluation methodology may also account for conflicting results reported in comparative research studies of OSCs [1,2,[4][5][6]11,12]. Despite appearing to share a similar evaluation approach (eg, using clinical case vignettes to compare OSCs to a 'gold standard' set by clinicians), there is notable variation in the methods used at various steps of the evaluation processes in these studies.…”