Particle-filter based audio-visual beat-tracking for music robot ensemble with human guitarist

Itohara,; Otsuka,; Mizumoto,; Ogata, Tetsuya; Okuno, Hiroshi G.

doi:10.1109/iros.2011.6048380

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notice that, the visual information is also very helpful when the auditory data is strongly corrupted by noise or by multiple sound sources. Another example can be found in [7] where the authors combine information coming from the two modalities to perform beat tracking of a person playing the guitar. Auditory and visual information is combined together to better address the problems of beat-tracking, tempo changes, and varying note lengths.…”

Section: Figmentioning

confidence: 99%

Online multimodal speaker detection for humanoid robots

Sanchez-Riera

Alameda-Pineda

Wienke

et al. 2012

2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012)

View full text Add to dashboard Cite

Abstract-In this paper we address the problem of audiovisual speaker detection. We introduce an online system working on the humanoid robot NAO. The scene is perceived with two cameras and two microphones. A multimodal Gaussian mixture model (mGMM) fuses the information extracted from the auditory and visual sensors and detects the most probable audio-visual object, e.g., a person emitting a sound, in the 3D space. The system is implemented on top of a platformindependent middleware and it is able to process the information online (17Hz). A detailed description of the system and its implementation are provided, with special emphasis on the online processing issues and the proposed solutions. Experimental validation, performed with five different scenarios, show that that the proposed method opens the door to robust humanrobot interaction scenarios.

show abstract

Section: Figmentioning

confidence: 99%

Online multimodal speaker detection for humanoid robots

Sanchez-Riera

Alameda-Pineda

Wienke

et al. 2012

2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012)

View full text Add to dashboard Cite

show abstract

“…Itohara et al [9] proposed an audio-visual beat-tracking method using both guitar sounds and guitarist's arm motions. They formulated a simplified model that represents a guitarist's arm trajectory as a sine wave, and integrated acoustic and skeleton features by using a state-space model.…”

Section: Audio-visual Beat Trackingmentioning

confidence: 99%

“…This enables us to use the Fourier transform. More specifically, the two signals y st j (t) and y tr j (t) corresponding to I st j and I tr j are given by y st (9) where N (x|µ, σ) represents a Gaussian function where x is a variable and parameters µ and σ correspond to the mean and standard deviation.…”

Section: Extraction Of Skeleton Featuresmentioning

confidence: 99%

Audio-visual beat tracking based on a state-space model for a music robot dancing with humans

Ohkita

Bando

Ikemiya

et al. 2015

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

This paper presents an audio-visual beat-tracking method for an entertainment robot that can dance in synchronization with music and human dancers. Conventional music robots have focused on either music audio signals or dancing movements of humans for detecting and predicting beat times in real time. Since a robot needs to record music audio signals by using its own microphones, however, the signals are severely contaminated with loud environmental noise and reverberant sounds. Moreover, it is difficult to visually detect beat times from real complicated dancing movements that exhibit weaker repetitive characteristics than music audio signals do. To solve these problems, we propose a state-space model that integrates both audio and visual information in a probabilistic manner. At each frame, the method extracts acoustic features (audio tempos and onset likelihoods) from music audio signals and extracts skeleton features from movements of a human dancer. The current tempo and the next beat time are then estimated from those observed features by using a particle filter. Experimental results showed that the proposed multi-modal method using a depth sensor (Kinect) for extracting skeleton features outperformed conventional mono-modal methods by 0.20 (F measure) in terms of beat-tracking accuracy in a noisy and reverberant environment.

show abstract

“…Interesting approaches that propose methods specifically conceived for humanoid robots are available, e.g. speech recognition (Nakadai et al, 2004), beat tracking (Itohara et al, 2011(Itohara et al, , 2012, active audition (Kim et al, 2007) or sound recognition (Nakamura et al, 2011), to cite a few. All of these methods deal with the detection and localization problems by using combinations of off-the-shelf techniques.…”

Section: Related Workmentioning

confidence: 99%

Vision-guided robot hearing

Alameda-Pineda

Horaud

2014

The International Journal of Robotics Research

View full text Add to dashboard Cite

Natural human-robot interaction in complex and unpredictable environments is one of the main research lines in robotics. In typical real-world scenarios, humans are at some distance from the robot and the acquired signals are strongly impaired by noise, reverberations and other interfering sources. In this context, the detection and localisation of speakers plays a key role since it is the pillar on which several tasks (e.g.: speech recognition and speaker tracking) rely. We address the problem of how to detect and localize people that are both seen and heard by a humanoid robot. We introduce a hybrid deterministic/probabilistic model. Indeed, the deterministic component allows us to map the visual information into the auditory space. By means of the probabilistic component, the visual features guide the grouping of the auditory features in order to form AV objects. The proposed model and the associated algorithm are implemented in real-time (17 FPS) using a stereoscopic camera pair and two microphones embedded into the head of the humanoid robot NAO. We performed experiments on (i) synthetic data, (ii) a publicly available data set and (iii) data acquired using the robot. The results we obtained validate the approach and encourage us to further investigate how vision can help robot hearing.

show abstract

Particle-filter based audio-visual beat-tracking for music robot ensemble with human guitarist

Cited by 3 publications

References 0 publications

Online multimodal speaker detection for humanoid robots

Online multimodal speaker detection for humanoid robots

Audio-visual beat tracking based on a state-space model for a music robot dancing with humans

Vision-guided robot hearing

Contact Info

Product

Resources

About