Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue

Abstract

Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.

Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

“…Although CVP is the more commonly used protocol [8,12,19,20,22], in this study CVP only identified approximately half of the distinct problems, whereas RVP identified three quarters. This contradicts the results of previous studies conducted by van den Haak et al [25][26][27][28], that compared the two verbal protocols with sighted participants and found that they were comparable in terms of effectiveness. One possible explanation as to why the results are different lies in what van den Haak et al identify as a user problem.…”

“…Comparing the two protocols in terms of whether they identify the same problems, we found that only 27 % of the distinct problems were identified by both protocols. Van den Haak et al [25][26][27][28]] also compared overlap between the two protocols in their studies. The overlap in most of the studies [25][26][27] was similar with the overlap reported in the study here, except for one study [28].…”

“…Van den Haak et al [25][26][27][28]] also compared overlap between the two protocols in their studies. The overlap in most of the studies [25][26][27] was similar with the overlap reported in the study here, except for one study [28]. Unfortunately, van den Haak et al did not specifically report the overlap between CVP and RVP, as they included other protocols in their studies.…”

“…A number of studies compared the number of problems revealed between the two protocols and most of them have demonstrated that the two protocols revealed a comparable set of problems [25][26][27][28]. However, these studies have some limitations, as only one website was used in each one of the studies.…”

Evaluating municipal websites: A methodological comparison of three think-aloud variants

“…As such, reactivity might affect the working of the concurrent think-aloud method. The extent to which this happens has been and continues to be a much investigated topic (Russo et al, 1989;Ericsson and Simon, 1993;Van den Haak, De Jong & Schellens, 2003;Van den Haak, De Jong & Schellens, 2004;Alavi, 2005).…”