Learning visual affordances of objects and tools through autonomous robot exploration

Abstract

Endowing artificial agents with the ability of predicting the consequences of their own actions and efficiently planning their behaviors based on such predictions is a fundamental challenge both in artificial intelligence and robotics. A computationally practical yet powerful way to model this knowledge, referred as objects affordances, is through probabilistic dependencies between actions, objects and effects: this allows to make inferences across these dependencies, such as i) predicting the effects of an action over an object, or ii) selecting the best action from a repertoire in order to obtain a desired effect over an object. We propose a probabilistic model capable of learning the mutual interaction between objects in complex tasks that involve manipulation, where one of the objects plays an active tool role while being grasped and used (e.g., a hammer) while another item is passively acted upon (e.g., a nail).We consider visual affordances, meaning that we do not model object labels or categories; instead, we compute a set of visual features that represent geometrical properties (e.g., convexity, roundness), which allows to generalize previouslyacquired knowledge to new objects. We describe an experiment in which a simulated humanoid robot learns an affordance model by autonomously exploring different actions with the objects present in a playground scenario. We report results showing that the robot is able to i) learn meaningful relationships between actions, tools, other objects and effects, and to ii) exploit the acquired knowledge to make predictions and take optimal decisions.

Affordances in Psychology, Neuroscience, and Robotics: A Survey

“…An important step toward solving this problem in a more general way, by considering the properties of both tools and affected objects and modeling how those properties influence the effects of specific actions, has been recently made by Gonçalves et al [221], [222]. In their work, the Bayesian Network (BN) probabilistic model initially proposed in [184] is extended to consider not just actions that are directly applied to an object, but also actions that involve the use of a tool, i.e.…”

Learning at the ends: From hand to tool affordances in humanoid robots

“…The features that we extract are pre-categorical shape descriptors computed as geometric relationships between perimeter, area, convex hull and approximated shapes of the segmented silhouettes of the objects in front of the robot, and they are based on [32]. Our visual feature extraction is similar to our previous works [19], [20], [22], however it now incorporates a richer set of 13 features instead of 5: convexity, eccentricity, compactness, circularity, squareness, number of convexity defects (i.e., number of cavities along the contour, for example the "holes" between fingers in a hand image), and seven central normalized moments. It is worth noting that we do not classify or label our shapes into classes, but we merely reason about their shape-derived features: this gives our system some flexibility, making it able to process unknown situations not seen during training.…”

“…Similar to previous research [12], [19], [20], we assume that the affordance of an object and of a robot manipulator is viewpoint-invariant. By exploiting this notion, it is possible to artificially augment the trials data using multiple views of manipulators and objects.…”

A deep probabilistic framework for heterogeneous self-supervised learning of affordances

“…Gonçalves et al [16], [17] have extended the Bayesian network formulation of [5] to incorporate the notion of tools. This model was further developed in [18] to allow the robot to generalize the learned affordances of its own hands to unseen tools.…”

“…This model was further developed in [18] to allow the robot to generalize the learned affordances of its own hands to unseen tools. In a scenario similar to [16] we proposed the usage of denoising auto-encoders to learn the affordances of objects and tools in continuous space [19]. Our model could use the continuous nature of features and effects while improving on the accuracy of previous works over different measures of performance.…”