Concurrent versus Retrospective Verbal Protocol for Comparing Window Usability

Abstract

A traditional concurrent verbal protocol method was compared to a heavily cued retrospective verbal protocol in which users were presented with a video tape of their performance to help them recall their thoughts after task completion. The two methods of protocol were employed in a comparison of two different size monitors. Subjects were required to complete 12 tasks which varied in the number of windows required simultaneously on the monitor. The subjects' performance, as measured by steps to completion, task completion time, and errors committed, was compared across monitors and protocol methods. Subjective data were also collected in the form of task difficulty ratings, as well as a global measure of user satisfaction. Verbal data were compared to assess any infonnation differences due to the methods of collection or the monitor sizes.No performance or subjective differences were found between the two protocol methods. The kinds of information gathered were quite different for the two methods, with concurrent protocol subjects giving procedural information and retrospective protocol subjects giving explanations and design statements. Performance data, as well as subjective data, indicated that on tasks that require that one or two windows be present simultaneously, there were no differences between the two monitor sizes. As the number of simultaneous windows increased, however, the large monitor's advantages became apparent. Tasks which require that four windows be present simultaneously were judged to be easier and required fewer steps on the large monitor than on the small monitor.

Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports

“…However, within usability testing research, the concurrent think-aloud and RTA have emerged as separate techniques. The concurrent technique is reported as being used more frequently by practitioners (McDonald, Edwards, and Zhao 2012), despite evidence that the retrospective technique generates the type of explanations and reflections that practitioners find to be particularly useful (Bowers and Snyder 1990;Ohnemus and Biers 1993;van den Haak, de Jong, and Schellens 2003). The speed and simplicity of concurrent elicitation coupled with the immediacy of feedback means that the concurrent technique is well suited to situations, such as usability testing, where results are often needed within a short timeframe.…”

“…The Retrospective Think-aloud (RTA), however, has received less attention (Guan et al 2006;Eger et al 2007;Elling, Lentz, and de Jong 2011). Elicitation procedures for the RTA frequently involve users providing their verbal protocols about tasks in a single block at the end of the test; usually cued by a muted video replay of the session (Bowers and Snyder 1990;Page and Rahimi 1995;van den Haak, de Jong, and Schellens 2003). However, the greater the interval between task completion and retrospective recall, the more likely it is that the accuracy of the RTA will suffer: users may simply forget the reasons for their behaviours, they may generalise across tasks or indeed rationalise their behaviours Simon 1984, 1993;Taylor and Dionne 2000).…”

Comparing Concurrent and Retrospective Verbal Protocols for Blind and Sighted Users

“…The thinking aloud may be performed concurrently with conducting the task, known as a concurrent verbal protocol (CVP), or retrospectively while reviewing recordings of their performance on a task, known as a retrospective verbal protocol (RVP). A number of studies have compared these two types of verbal protocol with sighted participants, in terms of the information gathered [3,17] and the number of problems revealed [25,26]. However, a comparison of these protocols when used with blind participants has not yet been performed, in spite of the fact that there are a number of studies which have used verbal protocols with blind participants [8,12,19,20,22].…”

“…Several studies have been conducted to compare the differences found in the information gathered between the two methods [3,10,17]. Bowers and Snyder [3] conducted a study comparing the two protocols in a multiple window task.…”

“…Bowers and Snyder [3] conducted a study comparing the two protocols in a multiple window task. Their results revealed that during CVP more procedural information was collected, whereas during RVP more design changes and explanations were collected.…”