Identification of High-Level Object Manipulation Operations from Multimodal Input

Abstract

Object manipulation constitutes a large part of our daily hand movements. Recognition of such movements by a robot in an interactive scenario is an issue that is rapidly gaining attention. In this paper we present an approach to identification of a class of high-level manual object manipulations. Experiments have shown that the naive approach based on classification of low-level sensor data yields poor performance. In this paper we introduce a two-stage procedure that considerably improves the identification performance. In the first stage of the procedure we estimate an intermediate representation by applying a linear preprocessor to the multimodal low-level sensor data. This mapping calculates shape, orientation and weight estimators of the interaction object. In the second stage we generate a classifier that is trained to identify high-level object manipulations given the intermediate representation based on shape, orientation and weight. The devices used in our procedure are: Immersion CyberGlove II enhanced with five tactile sensors on the fingertips (TouchGlove), nine tactile sensors to measure the change of the object's weight and a VICON multicamera system for trajectory recording. We have achieved the following recognition rates for 3600 data samples representing a sequence of manual object manipulations: 100% correct labelling of "holding", 97% of "pouring", 81% of "squeezing" and 65% of "tilting".

Hierarchical Bayesian Modeling of Manipulation Sequences from Bimodal Input

“…The assignment of product models to the bimanual tactile data in the first step has yielded 100 percent correct results. A more detailed evaluation of the segmentation procedure can be found in (Barchunova, Haschke, Grossekathoefer, et al, 2011).…”