We understand the environment by integrating information obtained by the senses of sight , hearing and touch. To integrate information across different senses, we must find the correspondence of events observed by different senses . This

No abstract

No abstract

Relating audiovisual events is important for constructing an artificial intelligent system, which can acquire the audiovisual knowledge of moving objects through active observation without a supervisor. This paper proposes a method for relating multiple audiovisual events observed by a camera and two microphones in the scene that a moving object is not in the view of the camera. As corresponding cues, we use Gestalt's grouping law : simultaneity of sound onsets and changes in movement, similarity of repetition between sound and movement, similarity of direction between sound and movement. Based on the correlation coefficient between sound onset series and STI time series, the component of frequency at sound onset is related to the spatiotemporal invariant sequence (STI sequence) of movement. We experimented in the real environment and obtained satisfactory results showing the effectiveness of the proposed method.