Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation

Abstract

Objective: To report on the use of an eye-tracking retrospective think-aloud for usability evaluation and to describe its application in assessing the usability of a mobile health app. Materials and Methods: We used an eye-tracking retrospective think-aloud to evaluate the usability of a HIV prevention mobile app among 20 young men (15–18 years) in New York City, NY; Birmingham, AL; and Chicago, IL. Task performance metrics, critical errors, a task completion rate per participant and a task completion rate per task, were measured. Eye-tracking metrics including fixation, saccades, time to first fixation, time spent, and revisits were measured and compared among participants with/without a critical error. Results: Using task performance analysis, we identified 19 critical errors on four activities, and of those, two activities had a task completion rate of less than 78%. To better understand these usability issues, we thoroughly analyzed participants’ corresponding eye movements and verbal comments using an in-depth problem analysis. In areas of interest created for the activity with critical usability problems, there were significant differences in time spent (p=0.008), revisits (p=0.004), and total numbers of fixations (p=0.007) by participants with/without a critical error. The overall mean score of perceived usability rated by the Health IT Usability Evaluation Scale was 4.64 (SD=0.33), reflecting strong usability of the app. Discussion and Conclusion: An eye-tracking retrospective think-aloud enabled us to identify critical usability problems as well as gain an in-depth understanding of the usability issues related to interactions between end-users and the app. Findings from this study highlight the utility of an eye-tracking retrospective think-aloud in consumer health usability evaluation research.

Human Factors and Organizational Issues

“…Cho et al, [16] demonstrated an improvement on usual eye-tracking protocols by devising a retrospective think-aloud, and related findings from eye tracking measures to success of failure in task completion. The new method asked participants to share their inner experience using the app and explain their experience and gaze patterns during errors making it possible to catch instances in which the user had more confidence than warranted, i.e., problems that would be undetected in traditional usability testing.…”

How Users' Gaze Behavior Is Related to Their Quality Evaluation of a Health Website Based on HONcode Principles?

“…For example, several studies have used eye-tracking techniques to examine how users interact with the web interface and put forward some suggestions for improving the interface design (Kules & Xie, 2011;Harris, Sillence, & Briggs, 2009). Besides, some researchers (Koybasi & Cagiltay, 2012;Cho et al, 2019;Schaarup et al, 2015) have used eye-tracking technology to evaluate the usability of online system, which increased the accuracy and completeness in the usability tests. However, in a systematic review, Maramba et al (2019) have identified, explored, and summarized the methods used in the usability testing of eHealth applications, and suggested that automated methods such as eye tracking have not gained traction in evaluating health apps.…”

A Visual Attentive Model for Discovering Patterns in Eye-Tracking Data—A Proposal in Cultural Heritage

“…A general attention model based on eye gaze was proposed. The use of an eye-tracking retrospective think-aloud for usability evaluation and description of this application in assessing the usability of a mobile health app is assessed in Reference [35]. The authors used an eye-tracking retrospective think-aloud to evaluate the usability of an HIV prevention mobile app among 20 young men (15-18 years) in New York City, NY; Birmingham, AL; and Chicago, IL.…”