Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers

Abstract

Symptom checkers are software tools that allow users to submit a set of symptoms and receive advice related to them in the form of a diagnosis list, health information or triage. The heterogeneity of their potential users and the number of different components in their user interfaces can make testing with end-users unaffordable. We designed and executed a two-phase method to test the respiratory diseases module of the symptom checker Erdusyk. Phase I consisted of an online test with a large sample of users (n=53). In Phase I, users evaluated the system remotely and completed a questionnaire based on the Technology Acceptance Model. Principal Component Analysis was used to correlate each section of the interface with the questionnaire responses, thus identifying which areas of the user interface presented significant contributions to the technology acceptance. In the second phase, the think-aloud procedure was executed with a small number of samples (n=15), focusing on the areas with significant contributions to analyze the reasons for such contributions. Our method was used effectively to optimize the testing of symptom checker user interfaces. The method allowed kept the cost of testing at reasonable levels by restricting the use of the think-aloud procedure while still assuring a high amount of coverage. The main barriers detected in Erdusyk were related to problems understanding time repetition patterns, the selection of levels in scales to record intensities, navigation, the quantification of some symptom attributes, and the characteristics of the symptoms.

Exploring Emergent Features of Student Interaction within an Embodied Science Learning Simulation

“…While students were prompted early in the interview to think aloud as much as possible, additional protocol reminders to consistently encourage thinking aloud were not explicit. This is consistent with the literature showing that a well-defined think-aloud protocol can lead to increased understanding between an instructor and a student, as well as elicit greater insights into the cognitive processes of learners as they interact with a system [48][49][50].…”

Digital and online symptom checkers and assessment services for urgent care to inform a new digital platform: a systematic review

“…Seven of these studies were evaluating symptom checkers as a self-contained intervention. 13,17,[19][20][21][22][23] Five of these studies were evaluating an e-consultation system that included a symptom checker 7,[31][32][33][34] and two of these studies were investigating public attitudes towards symptom checkers. 35,37 One study 17 reported that patients who were classified as less socioeconomically deprived were more likely to agree to use Internet Doctor than decline participation, although no relationship was found between socioeconomic deprivation and results in this study or between e-consultation use and socioeconomic deprivation in another study.…”

“…Seven cross-sectional papers were included in the review. 18,19,[35][36][37][38][39] Study characteristics are summarised in Table 13. The studies were carried out in the USA, 18,36,38,39 Norway 19,37 and Sweden 35 and all but one, a conference abstract, 38 were published articles.…”

“…All six papers included a clear statement of the research question and four of them presented clear outcome measures, but confounding variables were less clearly dealt with. Only two studies included a participant response rate, 19,35 and both of these indicated that the rate was > 50%, but only five studies recruited participants. Blinding of assessors was either not appropriate or not stated for all of the studies.…”

“…The process of navigating sites was reported as an issue that could increase frustration, for example if the pathway takes a long time to go through, the signposting is not clear and/or the entered information is not recognised as saved into the system. 19 Users were reported to be more likely to rush the task if the process was taking a long time, which increased the likelihood of self-diagnostic inaccuracy. 19 The ease of the task and the confidence in performing the task were associated with self-diagnostic accuracy; therefore, optimising usability appears to be an important aspect when designing a health information system.…”