What do usability test moderators say?

Abstract

Moderators in usability tests wrestle with the conflicting goals of obtaining relevant information from the users while at the same time avoiding to influence the users in ways that change how they use and feel about the tested system. In this study we investigate what moderators say by categorizing the moderator verbalizations from 12 test sessions. During the test tasks affirmations (38%) were the most common moderator verbalizations, followed by task instructions (32%) and prompts for reflection (16%). In addition, more of the moderator verbalizations during the tasks were closed (31%) than open (14%) and many more were positive (16%) than negative (1%). The moderators verbalized at a lower rate during the tasks than in the part of the sessions before the first task and after the last task. Still, they talked quite a lot. We discuss the content of their verbalizations and the implications of our findings.

Think-Aloud Verbalizations for Identifying User Experience Problems: Effects of Language Proficiency with Chinese Non-Native English Speakers

“…and the interactive TA protocol [12,31,46]) with those of native English speakers. Our findings show that for both non-native and native English participants, their verbalization categories and the relative proportions, the UX problems that they encountered, and the correlations between the verbalization categories and the UX problems were largely similar.…”

“…", and "keep talking" to encourage participants to think aloud. When conducting ITA sessions, the moderator actively probed participants using five types of prompts that were derived from prior studies [1,31,51]: Clarifying intentions ("What are you looking for? "), Seeking explanations ("Could you tell me why you did that?…”

“…Interactive Think-Aloud (ITA): In practice, it is not uncommon that the moderator actively asks participants questions during think-aloud usability test sessions to inquire about their opinions, explanations, or suggestions [12,31,46]. This variation of the concurrent TA protocols was often referred to as interactive think-aloud (ITA).…”