Attention-Enhanced Sensorimotor Object Recognition

Abstract

Sensorimotor learning, namely the process of understanding the physical world by combining visual and motor information, has been recently investigated, achieving promising results for the task of 2D/3D object recognition. Following the recent trend in computer vision, powerful deep neural networks (NNs) have been used to model the "sensory" and "motor" information, namely the object appearance and affordance. However, the existing implementations cannot efficiently address the spatio-temporal nature of the humanobject interaction. Inspired by recent work on attention-based learning, this paper introduces an attention-enhanced NN-based model that learns to selectively focus on parts of the physical interaction where the object appearance is corrupted by occlusions and deformations. The model's attention mechanism relies on the confidence of classifying an object based solely on its appearance. Three metrics are used to measure the latter, namely the prediction entropy, the average N-best likelihood difference, and the N-best likelihood dispersion. Evaluation of the attention-enhanced model on the SOR3D dataset reports 33% and 26% relative improvement over the appearance-only and the spatio-temporal fusion baseline models, respectively.

A deep learning approach to 2D/3D object affordance understanding

“…We develop three model variants to efficiently encode the spatio-temporal nature of the hand-object interaction, and investigate an attention mechanism that relies on the appearance stream confidence. The content of this chapter is based on the spatial and spatio-temporal models presented in [100] and [102], as well as on [101] where the attention mechanism is proposed.…”