Learning a Set of Interrelated Tasks by Using a Succession of Motor Policies for a Socially Guided Intrinsically Motivated Learner

Abstract

We aim at a robot capable to learn sequences of actions to achieve a field of complex tasks. In this paper, we are considering the learning of a set of interrelated complex tasks hierarchically organized. To learn this high-dimensional mapping between a continuous high-dimensional space of tasks and an infinite dimensional space of unbounded sequences of actions, we introduce a new framework called “procedures”, which enables the autonomous discovery of how to combine previously learned skills in order to learn increasingly complex combinations of motor policies. We propose an active learning algorithmic architecture, capable of organizing its learning process in order to achieve a field of complex tasks by learning sequences of primitive motor policies. Based on heuristics of active imitation learning, goal-babbling and strategic learning using intrinsic motivation, our algorithmic architecture leverages our procedures framework to actively decide during its learning process which outcome to focus on and which exploration strategy to apply. We show on a simulated environment that our new architecture is capable of tackling the learning of complex motor policies by adapting the complexity of its policies to the task at hand. We also show that our “procedures” enable the learning agent to discover the task hierarchy and exploit his experience of previously learned skills to learn new complex tasks.

Autonomous Reinforcement Learning of Multiple Interrelated Tasks

“…Trajectories with via points [18] and parametrised skills [19] are able to learn multiple motor trajectories, but this is commonly done considering single tasks/skills or assuming pre-defined tasks. Imitation learning has achieved important results in the learning of task hierarchies [20]- [22], even in association with IMs [23], but by definition it relies on external knowledge sources (i.e. the "instructor") thus limiting the autonomy of the systems.…”

CLIC: Curriculum Learning and Imitation for Object Control in Nonrewarding Environments

“…To our knowledge, very few works have tackled the issue of imitating third party agents that do not act as demonstrators. The focus has rather been on suboptimal demonstrations [16], [17] or identifying the best tutor among several ones [18].…”

“…Using absolute values ensures the agent will refocus on a task for which its competence drops. The interplay of curriculum and imitation learning in an autonomous agent is present in several works which also demonstrate that imitation learning can drive the skill acquisition trajectory of an autonomous agent learning from intrinsic motivations [30], [18]. However, these works address largely distinct technical issues and do not use deep RL.…”

“…After trajectories from Bob are observed, the agent imitates him for N imit steps (Figure 2, lines [18][19][20]. Adapting DQNfD [15] to the multi-goal case, we define a large margin classification loss for our general Q-values J g,w i I (Q) that writes…”

Sensorimotor contingencies as a key drive of development: from babies to robots

“…The work from Oudeyer et al (2005) confirms that a simple robot equipped with what has been called "intelligent adaptive curiosity" can indeed acquire information about its body, at least information of an implicit type --that is, the agent gradually learns to use its body more effectively to explore its environment. Following the idea that understanding one's effects on the environment is crucial for the autonomous development of animals and humans (White 1959;Berlyne, 1960) a variety of work in robotics has focused on the autonomous learning of skills on the basis of the interactions between the body of the artificial agent and the environment, where robots are tested in "simple" reaching or avoidance scenarios (e.g., Santucci et al, 2014a;Hafez et al, 2017;Reinhart, 2017;Hester and Stone, 2017;Tannenberg et al, 2019) or in more complex tasks involving interactions between objects (Da Silva, 2014;Seepanomwan et al, 2017), tool use (Forestier and Oudeyer, 2016) or hierarchical skill learning (Forestier et al, 2017;Colas et al, 2018;Santucci et al, 2019), and even in imitation learning experiments (Duminy et al, 2018). When combined with the use of "goals", intended here as specific states or effects that a system is willing to obtain, curiosity and intrinsic motivation are able to properly guide task selection (Merrick, 2012;Santucci et al, 2016) and reduce the exploration space (Rolf et al, 2010;Baranes and Oudeyer, 2013).…”