Deep Affordance-Grounded Sensorimotor Object Recognition

Abstract

It is well-established by cognitive neuroscience that human perception of objects constitutes a complex process, where object appearance information is combined with evidence about the so-called object "affordances", namely the types of actions that humans typically perform when interacting with them. This fact has recently motivated the "sensorimotor" approach to the challenging task of automatic object recognition, where both information sources are fused to improve robustness. In this work, the aforementioned paradigm is adopted, surpassing current limitations of sensorimotor object recognition research. Specifically, the deep learning paradigm is introduced to the problem for the first time, developing a number of novel neuro-biologically and neuro-physiologically inspired architectures that utilize state-of-the-art neural networks for fusing the available information sources in multiple ways. The proposed methods are evaluated using a large RGB-D corpus, which is specifically collected for the task of sensorimotor object recognition and is made publicly available. Experimental results demonstrate the utility of affordance information to object recognition, achieving an up to 29% relative error reduction by its inclusion.

Attention-Enhanced Sensorimotor Object Recognition

“…In this section, the object recognition problem based on hand-object interaction input is discussed. Notice that all models presented in the study utilize RGB-D sequences of the SOR3D dataset [6] captured by Kinect sensors (more details are provided in Section 4). A Fig.…”

“…A Fig. 1: Example video session "squeeze sponge" from SOR3D [6], sampled every 4 frames. The object appearance and the corresponding affordance information are presented as RGB frames.…”

“…On the other hand, recent studies in computer vision have adopted a more human-inspired approach, the so-called "sensorimotor learning" [4,5,6,7]. Cognitive neuroscience argues that human object perception is based on fusing the object appearance with its affordance [8,9], namely its functionalities or more specifically the set of actions that a human can perform with the object.…”

“…In fact, using NN-based models, the human-inspired two-stream information processing approach [15,16] can be modeled more efficiently. In particular, in our recent work [6], we investigated various two-stream fusion architectures to improve object recognition and achieved promising results. Besides sensorimotor learning, multi-stream fusion has been widely investigated in a variety of tasks, such as action recognition [17,18], audio-visual speech recognition [19,20], and voice activity detection [21].…”

“…In this paper, we investigate the integration of an attention mechanism within the sensorimotor object recognition framework. The spatio-temporal late fusion model of [6] is adopted, and the attention mechanism is integrated in the object appearance stream, prior to fusion. We argue that the object appearance features are the most informative for object recognition, while the affordance information should be attended when the appearance one is failing.…”