Pulling Back the Curtain on the Wizards of Oz

Abstract

The Wizard of Oz method is an increasingly common practice in HCI and CSCW studies as part of iterative design processes for interactive systems. Instead of designing a fully-fledged system, the 'technical work' of key system components is completed by human operators yet presented to study participants as if computed by a machine. However, little is known about how Wizard of Oz studies are interactionally and collaboratively achieved in situ by researchers and participants. By adopting an ethnomethodological perspective, we analyse our use of the method in studies with a voice-controlled vacuum robot and two researchers present. We present data that reveals how such studies are organised and presented to participants and unpack the coordinated orchestration work that unfolds 'behind the scenes' to complete the study. We examine how the researchers attend to participant requests and technical breakdowns, and discuss the performative, collaborative, and methodological nature of their work. We conclude by offering insights from our application of the approach to others in the HCI and CSCW communities for using the method.

The Last Decade of HCI Research on Children and Voice-based Conversational Agents

“…In this section we present the emergent research themes from our review, but first we cover a number of descriptive statistics. 3, most (𝑁 = 12) of the reviewed studies were quantitative in nature; seven of these used an experimental design and were done in a lab setting, where the participants interacted with a mock system through a Wizard of Oz (WoZ) approach [100] or a working prototype and remaining five developed quantitative models -including machinelearning models or regression models -using some form of corpus (e.g., stories told by children) or logs of child interactions with a system. In WoZ approach a human "wizard" simulates an intelligent system and interacts with a user through a real or mock interface [142].…”

It's Good to Talk: A Comparison of Using Voice Versus Screen-Based Interactions for Agent-Assisted Tasks

“…Porcheron et al [55]). Our agent, which we called Vizzy, was simulated by a human experimenter, who was tasked with triggering the prompts at certain times during the study.…”

Designing an Adaptive Embodied Conversational Agent for Health Literacy

“…Engagement is measured primarily using an adapted version of the short form User Engagement Scale (UES-SF) questionnaire. Our research questions are: 1 as a multi-platform system for running Wizard of Oz studies [29]. ALTCAI is based on the ARIA agent 2 , which uses the Greta ECA, and can deliver computer generated speech through the Text-To-Speech engine CereVoice 3 , text and a visual embodiment of a human with synchronised facial and bodily gestures.…”