Audio-visual beat tracking based on a state-space model for a music robot dancing with humans

Abstract

This paper presents an audio-visual beat-tracking method for an entertainment robot that can dance in synchronization with music and human dancers. Conventional music robots have focused on either music audio signals or dancing movements of humans for detecting and predicting beat times in real time. Since a robot needs to record music audio signals by using its own microphones, however, the signals are severely contaminated with loud environmental noise and reverberant sounds. Moreover, it is difficult to visually detect beat times from real complicated dancing movements that exhibit weaker repetitive characteristics than music audio signals do. To solve these problems, we propose a state-space model that integrates both audio and visual information in a probabilistic manner. At each frame, the method extracts acoustic features (audio tempos and onset likelihoods) from music audio signals and extracts skeleton features from movements of a human dancer. The current tempo and the next beat time are then estimated from those observed features by using a particle filter. Experimental results showed that the proposed multi-modal method using a depth sensor (Kinect) for extracting skeleton features outperformed conventional mono-modal methods by 0.20 (F measure) in terms of beat-tracking accuracy in a noisy and reverberant environment.

Audio-Visual Beat Tracking Based on a State-Space Model for a Robot Dancer Performing with a Human Dancer

“…Instead, the method keeps all the possibilities of tempos and beat times. If a unique tempo were extracted from music audio signals as in [15], tempo estimation failure would severely degrade the overall performance. We therefore formulate a nonlinear state-space model that has a tempo and a beat time as latent variables and acoustic and skeleton features as observed variables.…”

“…We compared the proposed audio-visual beat-tracking method with two conventional audio beat-tracking methods [5,15]. The method [5] is implemented in HARK [29] robot audition software, and its parameters are set to the default values except for m = 90.…”

“…The method [5] is implemented in HARK [29] robot audition software, and its parameters are set to the default values except for m = 90. The method [15] is similar to our method except that an audio tempo is uniquely determined in each frame as an acoustic feature. To evaluate the effectiveness of integrating the three kinds of features: onset likelihoods F k , audio tempo likelihoods R k (acoustic features), and visual tempo likelihoods S k (visual features), we tested an audio-based method using only F k and R k as well as a visual-based method using only F k and S k ( Table 1).…”

“…Such audio-visual integration has often been studied in the music information retrieval (MIR) community, and it has been shown to achieve better performance than singlemodal methods [9][10][11][12][13][14]. The proposed method is an improved version of our previous method [15]. To effectively integrate audio-visual information, it is necessary to extract intermediate features that represent the likelihood of a tempo and that of a beat time.…”

“…In each frame, we estimate the likelihood of each tempo and the onset likelihood of the current frame from music audio signals. This method is more advantageous than the previous method [15], which directly and uniquely estimates an audio tempo without allowing for other possibilities. On the other hand, another likelihood of each tempo is also calculated from skeleton information.…”