How to Reduce Computation Time While Sparing Performance During Robot Navigation? A Neuro-Inspired Architecture for Autonomous Shifting Between Model-Based and Model-Free Learning

Dromnelle, Rémi; Renaudo, Erwan; Pourcel, Guillaume; Chatila, Raja; Girard, Benoît; Khamassi, Mehdi

doi:10.1007/978-3-030-64313-3_8

Cited by 10 publications

(38 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another important current question is whether uncertainty alone is sufficient to arbitrate between MB and MF systems [49], or whether, when the two systems are equally uncertain, the agent should rely on the least computationally costly one [50]. If we want the agent to be initially agnostic about which system is more costly, and if we even want the agent to be able to potentially arbitrate between N different learning systems with different a priori unknown computational characteristics, then one proposal is simply to measure the average time taken by each system when it has to make decisions [50]. In some of the robotic experiments that we will describe in the next section, we found that this principle works robustly, enables to produce the ideal MB-to-MF sequence, not only during initial learning but also after a task change.…”

Section: Neuroscience Studies Of the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

“…A more recent series of robotic experiments with the same research goal (i.e., assessing the efficience and robustness of bio-inspired coordination principles of MB and MF learning) has been presented in [61,62,47] and later in [60,50,63]. First, [61,62,47] compared different coordination principles, including methods coming from ensemble learning [51] in several different simulated robotic tasks.…”

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

“…These experiments highlighted the respective advantages and disadvantages of MB and MF reinforcement learning in a variety of simulated robotic situations, and concluded again for the added value of coordinating them. In [60,50,63], simulated and real robotic experiments were presented, some involving navigation with a Turtlebot, and others involving simulated tasks with the PR2 robot and the Baxter robot (Fig. 3).…”

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

“…The last important result to highlight here is that an efficient coordination mechanism proposed by [50,63], and successfully applied to robot navigation and human-robot interaction scenarios, consists in taking into account not only the uncertainty but also the computational cost of each learning system. In practice, the proposed algorithm monitored the average time taken by each system to make its inference phase before deciding.…”

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

“…(Left) Human-Robot Interaction task tested in[60]: the human and the robot collaborate to put all boxes in a trashbin. (Right) Navigation task autonomously mapped and discretized by the robot during exploration[60,50]. The red area indicates the goal location whereas the green areas indicate starting locations of the robot.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Khamassi

2020

Theory and Practice of Natural Computing

Self Cite

View full text Add to dashboard Cite

Engineering approaches to machine learning (including robot learning) typically seek for the best learning algorithm for a particular problem, or a set problems. In contrast, the mammalian brain appears as a toolbox of different learning strategies, so that any newly encountered situation can be autonomously learned by an animal with a combination of existing learning strategies. For example, when facing a new navigation problem, a rat can either learn a map of the environment and then plan to find a path to its goal within this map. Alternatively, it can learn sequences of egocentric movements in response to identifiable features of the environment. For about 15 years, computational neuroscientists have searched for the mammalian brain's coordination mechanisms which enable it to find efficient, if not necessarily optimal, combinations of existing learning strategies to solve new problems. Understanding such coordination principles of multiple learning strategies could have great implications in robotics, to enable robots to autonomously determine which learning strategies are appropriate in different contexts. Here, we review some of the main neuroscience models for the coordination of learning strategies and present some of the early results obtained when applying these models to robot learning. We moreover highlight important energy costs which can be reduced with such bio-inspired solutions compared to current deep reinforcement learning approaches. We conclude by sketching a roadmap for further developing such bio-inspired hybrid learning approaches to robotics.

show abstract

Section: Neuroscience Studies Of the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

Section: Robotic Tests Of Bio-inspired Principles For the Coordination Of Model-based And Model-free Reinforcement Learningmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Khamassi

2020

Theory and Practice of Natural Computing

Self Cite

View full text Add to dashboard Cite

show abstract

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Dromnelle

Renaudo

Chétouani

et al. 2022

Int J of Soc Robotics

Self Cite

View full text Add to dashboard Cite

We present a new neuro-inspired reinforcement learning architecture for robot online learning and decision-making during both social and non-social scenarios. The goal is to take inspiration from the way humans dynamically and autonomously adapt their behavior according to variations in their own performance while minimizing cognitive effort. Following computational neuroscience principles, the architecture combines model-based (MB) and model-free (MF) reinforcement learning (RL). The main novelty here consists in arbitrating with a meta-controller which selects the current learning strategy according to a trade-off between efficiency and computational cost. The MB strategy, which builds a model of the long-term effects of actions and uses this model to decide through dynamic programming, enables flexible adaptation to task changes at the expense of high computation costs.The MF strategy is less flexible but also 1000 times less costly, and learns by observation of MB decisions. We test the architecture in three experiments: a navigation task in a real environment with task changes (wall configuration changes, goal location changes); a simulated object manipulation task under human teaching signals; and a simulated human-robot cooperation task to tidy up objects on a table. We show that our human-inspired strategy coordination method enables the robot to maintain an optimal performance in terms of reward and computational cost compared to an MB expert alone, which achieves the best performance but has the highest computational cost. We also show that the method makes it possible to cope with sudden changes in the environment, goal changes or changes in the behavior of the human partner during interaction tasks. The robots that performed these experiments, whether real or virtual, all used the same set of parameters, thus showing the generality of the method.

show abstract

Reinforcement-learning in fronto-striatal circuits

Averbeck

O’Doherty

2021

Neuropsychopharmacol.

View full text Add to dashboard Cite

How to Reduce Computation Time While Sparing Performance During Robot Navigation? A Neuro-Inspired Architecture for Autonomous Shifting Between Model-Based and Model-Free Learning

Cited by 10 publications

References 15 publications

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Reinforcement-learning in fronto-striatal circuits

Contact Info

Product

Resources

About