Interactively shaping robot behaviour with unlabeled human instructions

Najar, Anis; Sigaud, Olivier; Chétouani, Mohamed

doi:10.1007/s10458-020-09459-6

Cited by 15 publications

(52 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These algorithms will be presented in III-C. While other (more sophisticated) methods like Policy Shaping and Value Shaping [9], [10], [40], [45] to integrate Human feedback into the classical RL formulation exist, Reward Shaping will suffice as proof of concept for the feasibility of our approach. Ng et al [44] describe the necessary requirements for Reward Shaping to preserve the optimal policy, if these requirements are not met positivereward cycles can occur.…”

Section: A Interactive Rlmentioning

confidence: 99%

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Bied

Chétouani

2020

2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

Self Cite

View full text Add to dashboard Cite

An important aspect of Human-Robotcooperation is that the robot is capable of clearly communicating its intentions to its human collaborator. This communication of intentions often requires the generation of legible motion trajectories. The concept of legible motion is usually not studied together with machine learning. Studying these fields together is an important step towards better Human-Robot cooperation. In this paper, we investigate interactive robot learning approaches with the aim of developing models that are able to generate legible motions by taking observer feedback into account. We explore how to integrate the observer feedback into a Reinforcement Learning (RL) framework. We do this by proposing three different observer algorithms as observer strategies in an interactive RL scheme and compare with one non-interactive RL algorithm as baseline. For the observer strategies we vary the method how the observer estimates how likely the agent is going for the target goal. We evaluate our approach on five environments and calculate the legibility of the learned trajectories. The results show that the legibility of the learned trajectories is significantly higher while integrating the feedback from the observer compared with a standard Q-Learning algorithm not using the observer feedback.

show abstract

Section: A Interactive Rlmentioning

confidence: 99%

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Bied

Chétouani

2020

2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this way, new items can be actively loaded in WM and therefore the agent can be trained to maintain novel sensory representations on-the-fly [ 116 ]. Moreover, the keypresses used here minimize the efforts required to make arbitrary changes in the elicited stimuli [ 117 ]. Interactions as such provide external observers some time to interpret the behaviour of the robot, and form a real-time hypothesis of its forthcoming behaviour.…”

Section: Discussionmentioning

confidence: 99%

Embodied working memory during ongoing input streams

2021

View full text Add to dashboard Cite

Sensory stimuli endow animals with the ability to generate an internal representation. This representation can be maintained for a certain duration in the absence of previously elicited inputs. The reliance on an internal representation rather than purely on the basis of external stimuli is a hallmark feature of higher-order functions such as working memory. Patterns of neural activity produced in response to sensory inputs can continue long after the disappearance of previous inputs. Experimental and theoretical studies have largely invested in understanding how animals faithfully maintain sensory representations during ongoing reverberations of neural activity. However, these studies have focused on preassigned protocols of stimulus presentation, leaving out by default the possibility of exploring how the content of working memory interacts with ongoing input streams. Here, we study working memory using a network of spiking neurons with dynamic synapses subject to short-term and long-term synaptic plasticity. The formal model is embodied in a physical robot as a companion approach under which neuronal activity is directly linked to motor output. The artificial agent is used as a methodological tool for studying the formation of working memory capacity. To this end, we devise a keyboard listening framework to delineate the context under which working memory content is (1) refined, (2) overwritten or (3) resisted by ongoing new input streams. Ultimately, this study takes a neurorobotic perspective to resurface the long-standing implication of working memory in flexible cognition.

show abstract

“…The means by which teaching signals can be communicated to a learning agent vary. They can be provided via natural language (Kuhlmann et al, 2004 ; Cruz et al, 2015 ; Paléologue et al, 2018 ), computer vision (Atkeson and Schaal, 1997 ; Najar et al, 2020b ), hand-written programs (Maclin and Shavlik, 1996 ; Maclin et al, 2005a , b ; Torrey et al, 2008 ), artificial interfaces (Abbeel et al, 2010 ; Suay and Chernova, 2011 ; Knox et al, 2013 ), or physical interaction (Lozano-Perez, 1983 ; Akgun et al, 2012 ). Despite the variety of communication channels, we can distinguish two main categories of teaching signals based on how they are produced: advice and demonstration.…”

Section: Reinforcement Learning With Human Advicementioning

confidence: 99%

“…Despite the variety of communication channels, we can distinguish two main categories of teaching signals based on how they are produced: advice and demonstration. Even though advice and demonstration can share the same communication channels, like computer vision (Atkeson and Schaal, 1997 ; Najar et al, 2020b ) and artificial interfaces (Abbeel et al, 2010 ; Suay and Chernova, 2011 ; Knox et al, 2013 ), they are fundamentally different from each other in that demonstration requires the task to be executed by the teacher (demonstrated), while advice does not. In rare cases, demonstration (Whitehead, 1991 ; Lin, 1992 ) has been referred to as advice (Maclin and Shavlik, 1996 ; Maclin et al, 2005a ).…”

Section: Reinforcement Learning With Human Advicementioning

confidence: 99%

“…The interactive machine learning literature references a plethora of teaching signals such as instructions (Pradyot et al, 2012b ; Najar et al, 2020b ), demonstrations (Argall et al, 2009 ), and feedback (Knox and Stone, 2009 ; Najar et al, 2016 ). These signals can be categorized in several ways depending on what, when, and how they are produced.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning With Human Advice: A Survey

Najar

Chétouani

2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

In this paper, we provide an overview of the existing methods for integrating human advice into a reinforcement learning process. We first propose a taxonomy of the different forms of advice that can be provided to a learning agent. We then describe the methods that can be used for interpreting advice when its meaning is not determined beforehand. Finally, we review different approaches for integrating advice into the learning process.

show abstract

Interactively shaping robot behaviour with unlabeled human instructions

Cited by 15 publications

References 32 publications

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Integrating an Observer in Interactive Reinforcement Learning to Learn Legible Trajectories

Embodied working memory during ongoing input streams

Reinforcement Learning With Human Advice: A Survey

Contact Info

Product

Resources

About