An Embodied Agent Learning Affordances With Intrinsic Motivations and Solving Extrinsic Tasks With Attention and One-Step Planning

Abstract

We propose an architecture for the open-ended learning and control of embodied agents. The architecture learns action affordances and forward models based on intrinsic motivations and can later use the acquired knowledge to solve extrinsic tasks by decomposing them into sub-tasks, each solved with one-step planning. An affordance is here operationalized as the agent's estimate of the probability of success of an action performed on a given object. The focus of the work is on the overall architecture while single sensorimotor components are simplified. A key element of the architecture is the use of “active vision” that plays two functions, namely to focus on single objects and to factorize visual information into the object appearance and object position. These processes serve both the acquisition and use of object-related affordances, and the decomposition of extrinsic goals (tasks) into multiple sub-goals (sub-tasks). The architecture gives novel contributions on three problems: (a) the learning of affordances based on intrinsic motivations; (b) the use of active vision to decompose complex extrinsic tasks; (c) the possible role of affordances within planning systems endowed with models of the world. The architecture is tested in a simulated stylized 2D scenario in which objects need to be moved or “manipulated” in order to accomplish new desired overall configurations of the objects (extrinsic goals). The results show the utility of using intrinsic motivations to support affordance learning; the utility of active vision to solve composite tasks; and the possible utility of affordances for solving utility-based planning problems.

A computational model of language functions in flexible goal-directed behaviour

“…The remaining components, computationally simplified, are relevant to support the interaction of the agent with the simulated card environment (see 62 for the simple simulator used to this purpose). The visual sensor is formed by an RGBY matrix that encodes a small portion of the environment, approximately covering one card per time, and represents the input following an attention-focused saccade onto one specific card.…”

Autonomous Reinforcement Learning of Multiple Interrelated Tasks

“…The concept of Intrinsic Motivations (IMs) is borrowed from biological [9] and psychological literature [10] describing how novel or unexpected "neutral" stimuli, as well as the perception of control over the environment, can generate learning processes even in the absence of assigned rewards or tasks. In the computational literature, IMs have been implemented in artificial agents to foster their autonomy in gathering knowledge [11], [12], learning repertoire of skills [13], [14], [15], [16], exploiting affordances from the environment [17], [18], [19], selecting their own tasks [20], [21], [22], and even boosting imitation learning techniques [23].…”

A generative spiking neural-network model of goal-directed behaviour and one-step planning

“…One advantage, employed here, is that the world model can directly select actions to perform; instead, previous models [38,39] need an additional mechanism selecting actions on the basis of the state sequence produced by the world model. A second advantage is that for each environment state the world model can suggest the selection of actions that have a potential relevance in that context, rather than any action (this captures the popular idea of affordance in cognitive science [65,66]). A last advantage could be the easier learning (and understanding) of stateaction sequences directed to a goal produced by other agents; indeed, the world model would be neutral with respect to the fact that actions are performed by another part of the brain or by another agent.…”