Vero: A Method for Remotely Studying Human-AI Collaboration

Abstract

Despite the recognized need in the IS community to prepare for a future of human-AI collaboration, the technical skills necessary to develop and deploy AI systems are considerable, making such research difficult to perform without specialized knowledge. To make human-AI collaboration research more accessible, we developed a novel experimental method that combines a video conferencing platform, controlled content, and Wizard of Oz methods to simulate a group interaction with an AI teammate. Through a case study, we demonstrate the flexibility and ease of deployment of this approach. We also provide evidence that the method creates a highly believable experience of interacting with an AI agent. By detailing this method, we hope that multidisciplinary researchers can replicate it to more easily answer questions that will inform the design and development of future human-AI collaboration technologies.

Collective Intelligence in Human-AI Teams: A Bayesian Theory of Mind Approach

“…Furthermore, past work has often side-stepped challenges posed by language-based communication by constraining the scope to spatial or highly stylized tasks (Kaufmann, Gupta, and Taylor 2021;Baker et al 2017;Khalvati et al 2019). Others use Wizard of Oz techniques (Schelble et al 2022;Hohenstein et al 2022) to facilitate communication-based human-AI teaming interaction. To build an autonomous AI teammate that improves human-only team performance, one must build agents that overcome these obstacles.…”