Reinforcement Learning Via Practice and Critique Advice

Judah, Kshitij; Roy, Saikat; Fern, Alan; Dietterich, Thomas G.

doi:10.1609/aaai.v24i1.7690

Cited by 32 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A simulated human able to interact with the robot faces the table. We have defined two ways for the robot to learn from humans, drawing inspiration from the concepts of learning by evaluative feedback and learning by demonstration [Knox and Stone, 2009, Judah et al, 2010, Griffith et al, 2013. We name respectively the two types of underlying interventions: Intervention of the type congratulation and intervention of the type takeover.…”

Section: Simulated Humansmentioning

confidence: 99%

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Dromnelle

Renaudo

Chétouani

et al. 2022

Int J of Soc Robotics

View full text Add to dashboard Cite

We present a new neuro-inspired reinforcement learning architecture for robot online learning and decision-making during both social and non-social scenarios. The goal is to take inspiration from the way humans dynamically and autonomously adapt their behavior according to variations in their own performance while minimizing cognitive effort. Following computational neuroscience principles, the architecture combines model-based (MB) and model-free (MF) reinforcement learning (RL). The main novelty here consists in arbitrating with a meta-controller which selects the current learning strategy according to a trade-off between efficiency and computational cost. The MB strategy, which builds a model of the long-term effects of actions and uses this model to decide through dynamic programming, enables flexible adaptation to task changes at the expense of high computation costs.The MF strategy is less flexible but also 1000 times less costly, and learns by observation of MB decisions. We test the architecture in three experiments: a navigation task in a real environment with task changes (wall configuration changes, goal location changes); a simulated object manipulation task under human teaching signals; and a simulated human-robot cooperation task to tidy up objects on a table. We show that our human-inspired strategy coordination method enables the robot to maintain an optimal performance in terms of reward and computational cost compared to an MB expert alone, which achieves the best performance but has the highest computational cost. We also show that the method makes it possible to cope with sudden changes in the environment, goal changes or changes in the behavior of the human partner during interaction tasks. The robots that performed these experiments, whether real or virtual, all used the same set of parameters, thus showing the generality of the method.

show abstract

Section: Simulated Humansmentioning

confidence: 99%

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Dromnelle

Renaudo

Chétouani

et al. 2022

Int J of Soc Robotics

View full text Add to dashboard Cite

show abstract

“…Their systems have been shown to learn faster and with less feedback than other approaches. Interactive learning from demonstrations and instructions have also been shown to help teach different ways of behaving to a learning machine [86,88,[93][94][95][96][97].…”

Section: Interactive Approaches To Instruction Communication and Controlmentioning

confidence: 99%

Communicative capital: a key resource for human–machine shared agency and collaborative capacity

Mathewson

Parker

Sherstan

et al. 2022

Neural Comput & Applic

View full text Add to dashboard Cite

In this work, we present a perspective on the role machine intelligence can play in supporting human abilities. In particular, we consider research in rehabilitation technologies such as prosthetic devices, as this domain requires tight coupling between human and machine. Taking an agent-based view of such devices, we propose that human–machine collaborations have a capacity to perform tasks which is a result of the combined agency of the human and the machine. We introduce communicative capital as a resource developed by a human and a machine working together in ongoing interactions. Development of this resource enables the partnership to eventually perform tasks at a capacity greater than either individual could achieve alone. We then examine the benefits and challenges of increasing the agency of prostheses by surveying literature which demonstrates that building communicative resources enables more complex, task-directed interactions. The viewpoint developed in this article extends current thinking on how best to support the functional use of increasingly complex prostheses, and establishes insight toward creating more fruitful interactions between humans and supportive, assistive, and augmentative technologies.

show abstract

“…This is the general form of supervised learning. In RL, it includes labeling sets of actions as good or bad (Judah et al 2010;Christiano et al 2017), learning from demonstrations (Ross and Bagnell 2010;Abbeel and Ng 2004;Ho et al 2016), and corrections to dialogue agents (Li et al 2016;Chen et al 2017). In our setting, imperative feedback specifies a counterfactual behavior: something the learner should (or should not) have done (e.g., "You should have gone to the living room.").…”

Section: Extracting Mdp Features From Languagementioning

confidence: 99%

Learning Rewards From Linguistic Feedback

Sumers

Hawkins

et al. 2021

AAAI

View full text Add to dashboard Cite

We explore unconstrained natural language feedback as a learning signal for artificial agents. Humans use rich and varied language to teach, yet most prior work on interactive learning from language assumes a particular form of input (e.g., commands). We propose a general framework which does not make this assumption, instead using aspect-based sentiment analysis to decompose feedback into sentiment over the features of a Markov decision process. We then infer the teacher's reward function by regressing the sentiment on the features, an analogue of inverse reinforcement learning. To evaluate our approach, we first collect a corpus of teaching behavior in a cooperative task where both teacher and learner are human. We implement three artificial learners: sentiment-based "literal" and "pragmatic" models, and an inference network trained end-to-end to predict rewards. We then re-run our initial experiment, pairing human teachers with these artificial learners. All three models successfully learn from interactive human feedback. The inference network approaches the performance of the "literal" sentiment model, while the "pragmatic" model nears human performance. Our work provides insight into the information structure of naturalistic linguistic feedback as well as methods to leverage it for reinforcement learning.

show abstract

Reinforcement Learning Via Practice and Critique Advice

Cited by 32 publications

References 9 publications

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Communicative capital: a key resource for human–machine shared agency and collaborative capacity

Learning Rewards From Linguistic Feedback

Contact Info

Product

Resources

About