The Effects of System Initiative during Conversational Collaborative Search

Abstract

Our research in this paper lies at the intersection of collaborative and conversational search. We report on a Wizard of Oz lab study in which 27 pairs of participants collaborated on search tasks over the Slack messaging platform. To complete tasks, pairs of collaborators interacted with a so-called searchbot with conversational capabilities. The role of the searchbot was played by a reference librarian. It is widely accepted that conversational search systems should be able to engage in mixed-initiative interaction-take and relinquish control of a multi-agent conversation as appropriate. Research in discourse analysis differentiates between dialog-and task-level initiative. Taking dialog-level initiative involves leading a conversation for the sole purpose of establishing mutual belief between agents. Conversely, taking task-level initiative involves leading a conversation with the intent to influence the goals of the other agent(s). Participants in our study experienced three searchbot conditions, which varied based on the level of initiative the human searchbot was able to take:(1) no initiative, (2) only dialog-level initiative, and (3) both dialog-and task-level initiative. We investigate the effects of the searchbot condition on six different types of outcomes: (RQ1) perceptions of the searchbot's utility, (RQ2) perceptions of workload, (RQ3) perceptions of the collaboration, (RQ4) patterns of communication and collaboration, and perceived (RQ5) benefits and (RQ6) challenges from engaging with the searchbot. CCS Concepts: • Information systems → Collaborative search; • Human-centered computing → HCI design and evaluation methods.