Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots

Abstract

Much of NLP research has focused on crowdsourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training models on the conversations they have with humans in the game the models progressively improve, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect.

Reason first, then respond: Modular Generation for Knowledge-infused Dialogue

“…In the following experiments, we focus on the textbased open-world adventure game dialogue setting of LIGHT (Urbanek et al, 2019). More specifically, we consider LightWild (Shuster et al, 2021b), a dataset of more than 40k episodes which are not specifically knowledge grounded, but require commonsense reasoning and attention to detail of the context instead. Hence, we do not consider retrieval-augmented models for this task.…”

BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage

“…An alternative approach is to deploy a system publicly, and collect interaction data and feedback from organic users directly. The promise of such an approach is that the distribution of data will more closely match those organic users' desires, rather than decided by the researchers themselves when creating datasets (Gabriel et al, 2020;Shuster et al, 2021b;Ouyang et al, 2022). Further, continued deployment of such a system, with appropriate learning systems, could then potentially keep improving over time (Carlson et al, 2010;Agichtein et al, 2006;Madotto et al, 2021;Shuster et al, 2021b), where (Hancock et al, 2019) refer to this approach as a self-feeding chatbot.…”

“…The promise of such an approach is that the distribution of data will more closely match those organic users' desires, rather than decided by the researchers themselves when creating datasets (Gabriel et al, 2020;Shuster et al, 2021b;Ouyang et al, 2022). Further, continued deployment of such a system, with appropriate learning systems, could then potentially keep improving over time (Carlson et al, 2010;Agichtein et al, 2006;Madotto et al, 2021;Shuster et al, 2021b), where (Hancock et al, 2019) refer to this approach as a self-feeding chatbot. The challenge, however, is that organic users may not be invested enough to want to provide adequate feedback, and some may be adversarial (Park et al, 2021) as in the case of Microsoft's Tay (Davis, 2016).…”

“…Firstly, if conversations are relatively symmetric between conversational partners, the human side of the conversation can directly be used as a target for the model to mimic, which makes the learning algorithm straightforward. This was shown to give large improvements in the deployed LIGHT system (Shuster et al, 2021b). Such an approach is not directly applicable if the conversations are asymmetric, for example in the case of humans treating the bot like an assistant (whereas they do not want the bot to treat them like an assistant).…”

“…If successful, this could provide large-scale organic interactions with humans, and give the opportunity to study the continual improvement of models over time. Further, we expect innovation in this area will be accelerated if the artifacts of such a system are made available to the research community Shuster et al, 2021b).…”