Interactive Learning from Activity Description

Nguyen, Khanh; Misra, Dipendra; Schapire, Robert E.; Dudík, Miro; Shafto, Patrick

doi:10.48550/arxiv.2102.07024

Cited by 2 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rather than provide prescriptive advice, we explored having the coach provide advice by relabeling an agent's trajectory with the goal it achieved. We can then train this now-successful relabeled trajectory using supervised learning, as was done in [36]. However, we found that hindsight relabeling performed poorly, except on the simplest environments.…”

Section: J2 Hindsight Relabelingmentioning

confidence: 99%

Teachable Reinforcement Learning via Advice Distillation

Watkins¹,

Darrell²,

Abbeel³

et al. 2022

Preprint

View full text Add to dashboard Cite

Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can we overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based on "teachable" decision-making systems that learn from structured advice provided by an external teacher. We begin by formalizing a class of human-in-the-loop decision making problems in which multiple forms of teacher-provided advice are available to a learner. We then describe a simple learning algorithm for these problems that first learns to interpret advice, then learns from advice to complete tasks even in the absence of human supervision. In puzzle-solving, navigation, and locomotion domains, we show that agents that learn from advice can acquire new skills with significantly less human supervision than standard reinforcement learning algorithms and often less than imitation learning.Human learners, by contrast, leverage numerous, rich forms of supervision: joint attention [34], physical corrections [5] and natural language instruction [9]. For human teachers, this kind of 35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.

show abstract

Section: J2 Hindsight Relabelingmentioning

confidence: 99%

Teachable Reinforcement Learning via Advice Distillation

Watkins¹,

Darrell²,

Abbeel³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Moving to autotelic agents, similar processes could be transposed to infer a goal rather than a reward function. A few recent works start addressing this issue by captioning the goal of a demonstration through natural language and a goal generator Zhou & Small (2020); Nguyen et al (2021). We expect follow-up of such works to contribute to answering the key question of the nature of the information conveyed by the tutor through the task channel MacGlashan & Littman (2015); Ho et al (2016;.…”

Section: Inference From Goal-related Signalsmentioning

confidence: 99%

Towards Teachable Autotelic Agents

Sigaud¹,

Akakzia²,

Caselles-Dupré³

et al. 2021

Preprint

View full text Add to dashboard Cite

Autonomous discovery and direct instruction are two extreme sources of learning in children, but educational sciences have shown that intermediate approaches such as assisted discovery or guided play resulted in better acquisition of skills. When turning to Artificial Intelligence, the above dichotomy can be translated into the distinction between autonomous agents, which learn in isolation from their own signals, and interactive learning agents which can be taught by social partners but generally lack autonomy. In between should stand teachable autonomous agents: agents that learn from both internal and teaching signals to benefit from the higher efficiency of assisted discovery processes. Designing such agents could result in progress in two ways. First, very concretely, it would offer a way to non-expert users in the real world to drive the learning behavior of agents towards their expectations. Second, more fundamentally, it might be a key step to endow agents with the necessary capabilities to reach general intelligence. The purpose of this paper is to elucidate the key obstacles standing in the way towards the design of such agents. We proceed in four steps. First, we build on a seminal work of Bruner to extract relevant features of the assisted discovery processes happening between a child and a tutor. Second, we highlight how current research on intrinsically motivated agents is paving the way towards teachable and autonomous agents. In particular, we focus on autotelic agents, i.e. agents equipped with forms of intrinsic motivations that enable them to represent, self-generate and pursue their own goals. We argue that such autotelic capabilities from the learner side are key in the discovery process. Third, we adopt a social learning perspective on the interaction between a tutor and a learner to highlight some components that are currently missing to these agents before they can be taught by ordinary people using natural pedagogy. Finally, we provide a list of specific research questions that emerge from the perspective of extending these agents with assisted learning capabilities.

show abstract

Interactive Learning from Activity Description

Cited by 2 publications

References 29 publications

Teachable Reinforcement Learning via Advice Distillation

Teachable Reinforcement Learning via Advice Distillation

Towards Teachable Autotelic Agents

Contact Info

Product

Resources

About