Adaptive Coordination of Multiple Learning Strategies in Brains and Robots

Abstract

Engineering approaches to machine learning (including robot learning) typically seek for the best learning algorithm for a particular problem, or a set problems. In contrast, the mammalian brain appears as a toolbox of different learning strategies, so that any newly encountered situation can be autonomously learned by an animal with a combination of existing learning strategies. For example, when facing a new navigation problem, a rat can either learn a map of the environment and then plan to find a path to its goal within this map. Alternatively, it can learn sequences of egocentric movements in response to identifiable features of the environment. For about 15 years, computational neuroscientists have searched for the mammalian brain's coordination mechanisms which enable it to find efficient, if not necessarily optimal, combinations of existing learning strategies to solve new problems. Understanding such coordination principles of multiple learning strategies could have great implications in robotics, to enable robots to autonomously determine which learning strategies are appropriate in different contexts. Here, we review some of the main neuroscience models for the coordination of learning strategies and present some of the early results obtained when applying these models to robot learning. We moreover highlight important energy costs which can be reduced with such bio-inspired solutions compared to current deep reinforcement learning approaches. We conclude by sketching a roadmap for further developing such bio-inspired hybrid learning approaches to robotics.

The rodent lateral orbitofrontal cortex as an arbitrator selecting between model-based and model-free learning systems.

“…OFC dysfunction has been successfully modeled as an impairment in MB inferences resulting from disruption of the formation of latent states necessary for a detailed cognitive map of task space (Wilson et al, 2014). While this function effectively accounts for diverse experimental findings relating to the overall function of the OFC, we have recently found that dysfunction specific to the rodent lateral OFC causes a complex pattern of deficits in simple acquisition and extinction learning that is not clearly predicted by these RL theories (Panayi & Killcross, 2014, 2020. Here, we propose modifications to these RL models that can account for these findings.…”

“…This computational distinction also recently turned out useful in understanding how the balance between different learning strategies evolves through development (Decker et al, 2016), or how it varies between different individuals in Pavlovian conditioning paradigms, such as sign- versus goal-tracking behaviors (Cinotti, Marchand, et al, 2019; Lesaint et al, 2015). Finally, it is worth noting that this distinction is currently also a hot topic in machine learning and robotics (Khamassi, 2020; Kober et al, 2014; Wang et al, 2019), so that upcoming breakthroughs in these disciplines can later on fertilize computational neuroscience models of learning and decision-making.…”

Inscription corporelle des dynamiques cognitives et leur impact sur la liberté de l’humain en société

“…Il est à noter que cette stratégie sans modèle requière une longue phase d'apprentissage par essai-erreur et ne permet ensuite pas au robot de prédire les conséquences d'une action donnée. Mais l'intérêt réside dans le fait qu'une décision peut ensuite être rapide, évitant au robot d'effectuer les longs calculs requis s'il devait manipuler un modèle interne : le robot se contente de sélectionner le mouvement qui a la plus forte valeur à un moment donné de manière réactive en réponse à la reconnaissance de l'état de la tâche au moment de chaque décision (par exemple en reconnaissance d'un stimulus) (Khamassi, 2020a).…”

“…Une troisième propriété est intéressante à noter, car elle fait un lien avec le temps de délibération invoqué par Bergson : (3) un système sans modèle prend une décision très rapidement puisqu'il lui suffit de comparer un ensemble fini de valeurs (celles des actions en compétition dans la situation donnée) pour choisir l'action à la plus forte valeur (subjective). À l'opposé, un système avec modèle prend des décisions d'autant plus lentement qu'il lui faut estimer toutes les conséquences possibles d'une même action plusieurs coups à l'avance (par exemple lorsqu'on l'on joue aux échecs) et qu'il y a beaucoup d'actions possibles (Khamassi, 2020a).…”

Sequential Episodic Control

“…Studying the effects of imposing constraints on the memory capacity of episodic reinforcement learning models is something that previous approaches have not considered Ramani [2019], with the recent exception of Yalnizyan-Carson and Richards [2021]. The efficient use of a limited memory capacity is especially critical when embedding such algorithms in embodied systems such as robots that face strict computational and storage limitations Khamassi [2020] -a problem that human cognition also had to deal with Lisman and Idiart [1995], Jensen and Lisman [2001].…”