Coarse speech recognition by audio-visual integration based on missing feature theory

Trends in Applied Intelligent Systems

2010

Self Cite

Abstract. Noise-robust Automatic Speech Recognition (ASR) is essential for robots which are expected to communicate with humans in a daily environment. In such an environment, Voice Activity Detection (VAD) strongly affects the performance of ASR because there are many acoustically and visually noises. In this paper, we improved Audio-Visual VAD for our two-layered audio visual integration framework for ASR by using hangover processing based on erosion and dilation. We implemented proposed method to our audio-visual speech recognition system for robot. Empirical results show the effectiveness of our proposed method in terms of VAD.

Section: The Second Layer Av Integration Blockmentioning

confidence: 99%

An Improvement in Audio-Visual Voice Activity Detection for Automatic Speech Recognition

Trends in Applied Intelligent Systems

2010

Self Cite

“…We simply introduced our reported AVSR for robots [6] as mentioned in Section I, because this AVSR system showed high noise-robustness to improve speech recognition even when either audio or visual information is missing and/or contaminated by noises. This kind of high performance is derived from missing feature theory (MFT) which drastically improves noise-robustness by using only reliable acoustic and visual features by masking unreliable ones out.…”

Section: B the Second Layer Av Integration Blockmentioning

confidence: 99%

“…To solve the difficulties, we reported AVSR for robots by introducing two psychologically-inspired methods [6]. One is missing feature theory (MFT) which improves noise-robustness by using only reliable acoustic and visual features by masking unreliable ones out.…”

Section: Introductionmentioning

confidence: 99%

Automatic speech recognition improved by two-layered audio-visual integration for robot audition

2009 9th IEEE-RAS International Conference on Humanoid Robots

2009

Self Cite

Abstract-The robustness and high performance of ASR is required for robot audition, because people usually speak to each other to communicate. This paper presents two-layered audio-visual integration to make automatic speech recognition (ASR) more robust against speaker's distance and interfering talkers or environmental noises. It consists of Audio-Visual Voice Activity Detection (AV-VAD) and Audio-Visual Speech Recognition (AVSR). The AV-VAD layer integrates several AV features based on a Bayesian network to robustly detect voice activity, or speaker's utterance duration. This is because the performance of VAD strongly affects that of ASR. The AVSR layer integrates the reliability estimation of acoustic features and that of visual features by using a missing-feature theory method. The reliability of audio features is more weighted in a clean acoustic environment, while that of visual features is more weighted in a noisy environment. This AVSR layer integration can cope with dynamically-changing environments in acoustics or vision. The proposed AV integrated ASR is implemented on HARK, our open-sourced robot audition software, with an 8 ch microphone array. Empirical results show that our system improves 9.9 and 16.7 points of ASR results with/without microphone array processing, respectively, and also improves robustness against several auditory/visual noise conditions.

“…To tackle with these difficulties, we reported AVSR for a robot based on two psychologically-inspired methods [6]. One is Missing Feature Theory (MFT), which improves noise-robustness by using only reliable audio and visual features by masking unreliable ones out.…”

Section: Introductionmentioning

confidence: 99%

Two-layered audio-visual speech recognition for robots in noisy environments

2010 IEEE/RSJ International Conference on Intelligent Robots and Systems

2010

Abstract-Audio-visual (AV) integration is one of the key ideas to improve perception in noisy real-world environments. This paper describes automatic speech recognition (ASR) to improve human-robot interaction based on AV integration. We developed AV-integrated ASR, which has two AV integration layers, that is, voice activity detection (VAD) and ASR. However, the system has three difficulties: 1) VAD and ASR have been separately studied although these processes are mutually dependent, 2) VAD and ASR assumed that high resolution images are available although this assumption never holds in the real world, and 3) an optimal weight between audio and visual stream was fixed while their reliabilities change according to environmental changes. To solve these problems, we propose a new VAD algorithm taking ASR characteristics into account, and a linear-regression-based optimal weight estimation method. We evaluate the algorithm for auditoryand/or visually-contaminated data. Preliminary results show that the robustness of VAD improved even when the resolution of the images is low, and the AVSR using estimated stream weight shows the effectiveness of AV integration.