Joint Object Affordance Reasoning and Segmentation in RGB-D Videos

Abstract

The authors would like to thank the European Commission for financial support through H2020 project HR-Recycler under contract 820742 and Nvidia Corporation for a Titan X GPU donation ABSTRACT Understanding human-object interaction is a fundamental challenge in computer vision and robotics. Crucial to it is the ability to infer "object affordances" from visual data, namely the types of interaction supported by an object of interest and the object parts involved. Such inference can be approached as an "affordance reasoning" task, where object affordances are recognized and localized as image heatmaps, and as an "affordance segmentation" task, where affordance labels are obtained at a more detailed, image pixel level. To tackle the two tasks, existing methods typically: (i) treat them independently;(ii) adopt static image-based models, ignoring the temporal aspect of human-object interaction; and / or (iii) require additional strong supervision concerning object class and location. In this paper, we focus on both tasks, while addressing all three aforementioned shortcomings. For this purpose, we propose a deep-learning based dual encoder-decoder model for joint affordance reasoning and segmentation, which learns from our recently introduced SOR3D-AFF corpus of RGB-D human-object interaction videos, without relying on object localization and classification. The basic components of the model comprise: (i) two parallel encoders that capture spatio-temporal interaction information; (ii) a reasoning decoder that predicts affordance heatmaps, assisted by an affordance classifier and an attention mechanism; and (iii) a segmentation decoder that exploits the predicted heatmap to yield pixel-level affordance segmentation. All modules are jointly trained, while the system can operate on both static images and videos. The approach is evaluated on four datasets, surpassing the current state-of-the-art in both affordance reasoning and segmentation.

Human Hands as Probes for Interactive Object Understanding

“…Researchers have used other forms of supervision (strong supervision, weak supervision, imitation learning, reinforcement learning, inverse reinforcement learning) to build interactive understanding of objects. This can be in the form of learning a) where and how to grasp [9,21,26,27,30,35,36,39,43,51], b) state classifiers [25], c) interaction hotspots [15,42,44,61], d) spatial priors for action sites [46], e) object articulation modes [12,38], f) reward functions [29,31,50,52], g) functional correspondences [34]. While our work pursues similar goals, we differ in our supervision source (observation of human hands interacting with objects in egocentric videos).…”