An Empirical Study of the Effects of Three Think-Aloud Protocols on Identification of Usability Problems

Abstract

Think-aloud is a de-facto standard in user-based usability evaluation to verbalize what a user is experiencing. Despite its qualities, it has been argued that thinking aloud affects the task solving process. This paper reports from an empirical study of the effect of three think-aloud protocols on the identified usability problems. The three protocols were traditional, active listening and coaching. The study involved 43 test subjects distributed on the three thinkaloud conditions and a silent control condition in a between-subject design. The results show that the three think-aloud protocols facilitated identification of the double number of usability problems compared to the silent condition, while the problems identified by the three think-aloud protocol were comparable. Our results do not support the common emphasis on the Coaching protocol, while we have seen that the Traditional protocol performs surprisingly well.

Usability Problems Experienced by Different Groups of Skilled Internet Users: Gender, Age, and Background

“…Across the analysis, the evaluators had an any-two agreement of 0.44 (SD = 0.11), which is relatively high compared to other studies [3]. Further information about the data collection can be found in [1].…”

Manifestations of Users’ Privacy Concerns in a Formative Usability Test of Social Networking Site

“…The interactive and relaxed think-aloud protocol applied in the study is the most used protocol in usability testing [21]. The chosen think-aloud protocol affects very little on the number and type of usability problems found [24]. Therefore, we must emphasize that, in this setting, the test participants brought up the privacy issues naturally and spontaneously in discussion about system features and usability.…”

What do usability test moderators say?

“…Olmsted-Hawala et al [25] found that relaxed thinking aloud yielded higher satisfaction with the website than classic thinking aloud and higher task completion rates than both classic thinking aloud and the control condition. Bruun and Stage [4] found that classic and relaxed thinking aloud revealed more usability problems than the control condition; there was no difference between the two thinking-aloud conditions. These authors note that thinking aloud "led to data from which we could enrich problem descriptions to include notions of why a particular problem was observed.…”