Audio-visual robot command recognition

Abstract

This paper addresses the problem of audio-visual command recognition in the framework of the D-META Grand Challenge 1 . Temporal and non-temporal learning models are trained on visual and auditory descriptors. In order to set a proper baseline, the methods are tested on the "Robot Gestures" scenario of the publicly available RAVEL data set, following the leave-one-out cross-validation strategy. The classification-level audio-visual fusion strategy allows for compensating the errors of the unimodal (audio or vision) classifiers. The obtained results (an average audio-visual recognition rate of almost 80%) encourage us to investigate on how to further develop and improve the methodology described in this paper.

Benchmarking methods for audio-visual recognition using tiny training sets

“…The method is tested on a dataset of 900 videos and 12 classes. Similarly, a convex function is used in [6] to combine the unimodal classification. In this case the dataset consists of more than 200 sequences from 9 different classes.…”

“…This quantity is prohibitive for useradaptive methods whose discriminative power should be high when trained on tiny datasets (10-15 instances per class). Both [6,7] deal with such datasets. Albeit, the work in [7] uses sequential forward feature selection, an iterative algorithms that slows down the training process.…”