Relating Audio-Visual Events Caused by Multiple Movements: In the Case that a Moving Object is Out of Camera View

Abstract

Relating audiovisual events is important for constructing an artificial intelligent system, which can acquire the audiovisual knowledge of moving objects through active observation without a supervisor. This paper proposes a method for relating multiple audiovisual events observed by a camera and two microphones in the scene that a moving object is not in the view of the camera. As corresponding cues, we use Gestalt's grouping law : simultaneity of sound onsets and changes in movement, similarity of repetition between sound and movement, similarity of direction between sound and movement. Based on the correlation coefficient between sound onset series and STI time series, the component of frequency at sound onset is related to the spatiotemporal invariant sequence (STI sequence) of movement. We experimented in the real environment and obtained satisfactory results showing the effectiveness of the proposed method.