CraftAssist: A Framework for Dialogue-enabled Interactive Agents

Gray, Jonathan; Srinet, Kavya; Jernite, Yacine; Yu, Haonan; Chen, Zhuoyuan; Guo, Demi; Goyal, Siddharth; Zitnick, C. Lawrence; Szlam, Arthur

doi:10.48550/arxiv.1907.08584

Cited by 12 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In robotics, this might translate to building systems for cooking, perhaps taking inspiration from Epic Kitchens (Damen et al, 2018), where the set of high-level objectives (general recipes to follow, kitchen behaviors to imitate) is much larger, but where individual subtasks (low-level abstractions like slicing a vegetable, stirring a pot) are very common and generalizable. Other settings might include open-ended building tasks, either in the real world (Knepper et al, 2013;Lee et al, 2019), or in virtual worlds like Minecraft (Johnson et al, 2016;Gray et al, 2019).…”

Section: Discussion and Lessons Learnedmentioning

confidence: 99%

Learning Adaptive Language Interfaces through Decomposition

Karamcheti

Sadigh

Liang

2020

Preprint

View full text Add to dashboard Cite

Our goal is to create an interactive natural language interface that efficiently and reliably learns from users to complete tasks in simulated robotics settings. We introduce a neural semantic parsing system that learns new high-level abstractions through decomposition: users interactively teach the system by breaking down high-level utterances describing novel behavior into low-level steps that it can understand. Unfortunately, existing methods either rely on grammars which parse sentences with limited flexibility, or neural sequence-to-sequence models that do not learn efficiently or reliably from individual examples. Our approach bridges this gap, demonstrating the flexibility of modern neural systems, as well as the one-shot reliable generalization of grammar-based methods. Our crowdsourced interactive experiments suggest that over time, users complete complex tasks more efficiently while using our system by leveraging what they just taught. At the same time, getting users to trust the system enough to be incentivized to teach high-level utterances is still an ongoing challenge. We end with a discussion of some of the obstacles we need to overcome to fully realize the potential of the interactive paradigm.

show abstract

Section: Discussion and Lessons Learnedmentioning

confidence: 99%

Learning Adaptive Language Interfaces through Decomposition

Karamcheti

Sadigh

Liang

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Furthermore, specific environments have been created to present rich interaction contexts to participants while keeping the interaction channel fixed, usually in a turn-by-turn setting. An example is the Minecraft virtual environment in which communication happens between a player and a bot to build objects together (Gray et al, 2019). A previous version of the slurk software (Schlangen et al, 2018) has already been used for research, recruiting participants via AMT.…”

Section: Related Workmentioning

confidence: 99%

The slurk Interaction Server Framework: Better Data for Better Dialog Models

Götze¹,

Paetzel-Prüsmann²,

Liermann³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents the slurk software, a lightweight interaction server for setting up dialog data collections and running experiments. slurk enables a multitude of settings including text-based, speech and video interaction between two or more humans or humans and bots, and a multimodal display area for presenting shared or private interactive context. The software is implemented in Python with an HTML and JAVASCRIPT frontend that can easily be adapted to individual needs. It also provides a setup for pairing participants on common crowdworking platforms such as Amazon Mechanical Turk and some example bot scripts for common interaction scenarios.

show abstract

“…As a result, we plan to share resources, including a website, a community forum, a technical stack, and outreach efforts, which should make both competitions more effective at their individual goals. Besides MineRL Diamond, there have been other benchmarks that use Minecraft as a platform, such as MARL Ö , CraftAssist [Gray et al, 2019], Malmö [Johnson et al, 2016], and Generative Design in Minecraft (GDMC) [Salge et al, 2018], but none are particularly relevant to LfHF.…”

Section: Noveltymentioning

confidence: 99%

The MineRL BASALT Competition on Learning from Human Feedback

Shah¹,

Wild²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve.The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations.

show abstract

CraftAssist: A Framework for Dialogue-enabled Interactive Agents

Cited by 12 publications

References 0 publications

Learning Adaptive Language Interfaces through Decomposition

Learning Adaptive Language Interfaces through Decomposition

The slurk Interaction Server Framework: Better Data for Better Dialog Models

The MineRL BASALT Competition on Learning from Human Feedback

Contact Info

Product

Resources

About