Finding the Correspondence of Audio-Visual Events by Object Manipulation

Abstract

A human understands the objects in an environment by integrating information obtained by the senses of sight, hearing, and touch. In such integration, active movement plays an important role. We propose a method for determining the correspondence of audiovisual events by handling an object. The method uses the general grouping rules in Gestalt psychology, i.e.``simultaneity" and ``similarity" among motion commands, sound onsets, and motion of objects in images. This system comprises four components, motor, audio, visual, and integration (Fig. 1). In the motor part (Fig. 1 (1)), a computer sends motor commands to the manipulator to handle an unknown object. Then the object emits a sound and causes movement change in images. In the sound part (Fig. 1 (2)), sound onset time series are detected, and similar spectra at each sound onset are grouped into the same sound source of the object. In the visual part (Fig. 1 (3)), moving-object areas and its movements are extracted. The motion loci of object areas are calculated and the changes in object movement are recorded. In the integration part(Fig. 1 (4)), sampling rates of manipulator commands, audio signals, and camera images are converted to the same rate by re-sampling. Finally, we calculate correlations among audio, visual, and motor signals, and events with high correlation are grouped together. In experiments, we used a microphone, a camera, and a robot featuring a hand manipulator. The robot grasps an object like a bell and shakes it, or grasps an object like a stick and beats a drum with it. Those motions are periodic or non-periodic. Then the object emits periodical/non-periodical events. To create a more realistic scenario, we put an another event source (a metronome) in the environment. We conducted two trials for each of 40

Finding the correspondence of audiovisual events by object manipulation

“…We extracted acoustic information by approaching objects actively, and found the correspondence between motion and audiovisual observation by using the grouping factors of Gestalt psychology [14,15]. Approaching objects actively allows extraction of the objects' features; furthermore, the correspondence of audiovisual events caused by self-motion can be determined by motor information, even in environments involving multiple audiovisual events [16].…”

Unsupervised Learning of Central Cases of Audio-Visual Events

“…Consequently, the effectiveness of this method was confirmed. 画像と音信号から，段階的に相手と共有可能な概念を獲得 する研究も行われている (14) 。しかし，この研究では，物 体は単純な形をしており，システムと人が対話することに よって，物体について教示する必要がある。 我々は，運動情報，音の発生，物体の映像上での動きに 対して，同時性と類似性の手掛かりを用いて，能動的に視 聴覚事象を対応付ける手法を示した (15) 。また，対象物へ選 択的に注意を向けることで視聴覚事象を対応付ける手法を 示した (16)…”