Determining Correspondences Between Sensory and Motor Signals

Nishibori, Kento; Chen, Jinji; Takeuchi, Yoshinori; Matsumoto, Tetsuya; Kumakura, Hiroaki; Ohnishi, Noboru

doi:10.1007/978-3-540-30541-5_65

Cited by 3 publications

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We extracted acoustic information by approaching objects actively, and found the correspondence between motion and audiovisual observation by using the grouping factors of Gestalt psychology [14,15]. Approaching objects actively allows extraction of the objects' features; furthermore, the correspondence of audiovisual events caused by self-motion can be determined by motor information, even in environments involving multiple audiovisual events [16].…”

Section: Introductionmentioning

confidence: 99%

Finding the correspondence of audiovisual events by object manipulation

Nishibori

Takeuchi

Matsumoto

et al. 2009

Elect Comm in Japan

Self Cite

View full text Add to dashboard Cite

SUMMARYA human being understands the objects in the environment by integrating information obtained by the senses of sight, hearing, and touch. In this integration, active manipulation of objects plays an important role. We propose a method for finding the correspondence of audiovisual events by manipulating an object. The method uses the general grouping rules in Gestalt psychology, that is, "simultaneity" and "similarity" among motion command, sound onsets, and motion of the object in images. In experiments, we used a microphone, a camera, and a robot which has a hand manipulator. The robot grasps an object like a bell and shakes it or grasps an object like a stick and beats a drum in a periodic, or nonperiodic motion. Then the object emits periodic/nonperiodic events. To create a more realistic scenario, we put another event source (a metronome) in the environment. As a result, we had a success rate of 73.8% in finding the correspondence between audiovisual events (afferent signal) which are related to robot motion (efferent signal).

show abstract

Section: Introductionmentioning

confidence: 99%

Finding the correspondence of audiovisual events by object manipulation

Nishibori

Takeuchi

Matsumoto

et al. 2009

Elect Comm in Japan

Self Cite

View full text Add to dashboard Cite

show abstract

Finding the Correspondence of Audio-Visual Events by Object Manipulation

et al. 2008

View full text Add to dashboard Cite

A human understands the objects in an environment by integrating information obtained by the senses of sight, hearing, and touch. In such integration, active movement plays an important role. We propose a method for determining the correspondence of audiovisual events by handling an object. The method uses the general grouping rules in Gestalt psychology, i.e.``simultaneity" and ``similarity" among motion commands, sound onsets, and motion of objects in images. This system comprises four components, motor, audio, visual, and integration (Fig. 1). In the motor part (Fig. 1 (1)), a computer sends motor commands to the manipulator to handle an unknown object. Then the object emits a sound and causes movement change in images. In the sound part (Fig. 1 (2)), sound onset time series are detected, and similar spectra at each sound onset are grouped into the same sound source of the object. In the visual part (Fig. 1 (3)), moving-object areas and its movements are extracted. The motion loci of object areas are calculated and the changes in object movement are recorded. In the integration part(Fig. 1 (4)), sampling rates of manipulator commands, audio signals, and camera images are converted to the same rate by re-sampling. Finally, we calculate correlations among audio, visual, and motor signals, and events with high correlation are grouped together. In experiments, we used a microphone, a camera, and a robot featuring a hand manipulator. The robot grasps an object like a bell and shakes it, or grasps an object like a stick and beats a drum with it. Those motions are periodic or non-periodic. Then the object emits periodical/non-periodical events. To create a more realistic scenario, we put an another event source (a metronome) in the environment. We conducted two trials for each of 40

show abstract