Finding the correspondence of audiovisual events by object manipulation

Abstract

SUMMARYA human being understands the objects in the environment by integrating information obtained by the senses of sight, hearing, and touch. In this integration, active manipulation of objects plays an important role. We propose a method for finding the correspondence of audiovisual events by manipulating an object. The method uses the general grouping rules in Gestalt psychology, that is, "simultaneity" and "similarity" among motion command, sound onsets, and motion of the object in images. In experiments, we used a microphone, a camera, and a robot which has a hand manipulator. The robot grasps an object like a bell and shakes it or grasps an object like a stick and beats a drum in a periodic, or nonperiodic motion. Then the object emits periodic/nonperiodic events. To create a more realistic scenario, we put another event source (a metronome) in the environment. As a result, we had a success rate of 73.8% in finding the correspondence between audiovisual events (afferent signal) which are related to robot motion (efferent signal).