Audio-visual keyword spotting based on adaptive decision fusion under noisy conditions for human-robot interaction

Liu, Hong; Fan, Ting; Wu, Pingping

doi:10.1109/icra.2014.6907840

Cited by 13 publications

(8 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Filler models are based on context independent phoneme or viseme, which is modeled with a 3-state HMM. Since the performance of conventional cascade strategy which performs visual re-scoring on the acoustic hypothesis in acoustic noisy conditions drops significantly, a parallel strategy is proposed to complementarily make the best use of two modalities as illustrated in Fig.5 [22]. Acoustic keyword searching and visual keyword searching is first conducted in parallel, generating acoustic keyword candidates and visual keyword candidates with corresponding log-likelihoods.…”

Section: Adaptive Decision Fusionmentioning

confidence: 99%

Audio-visual Keyword Spotting for Mandarin Based on Discriminative Local Spatial-Temporal Descriptors

Liu¹,

Fan²,

Wu³

2014

2014 22nd International Conference on Pattern Recognition

Self Cite

View full text Add to dashboard Cite

Although keyword spotting (KWS) technologies have been successfully applied to some applications, most KWS systems have a common problem of noise-robustness when applied to real-world environments. Audio-visual keyword spotting (AVKWS) using both acoustic and visual information is a solution to complementarily solve the problem. Most existing audio-visual speech recognition (AVSR) systems extract geometric features as visual features, which heavily rely on accurate and reliable detection and tracking of facial feature points. To avoid this defect of geometric features, an appearance-based discriminative local spatial-temporal descriptor (disCLBP-TOP) is proposed in this paper, which devotes to extracting robust and discriminative patterns of interest. Besides, a parallel two-step recognition based on both acoustic and visual keyword searching and re-scoring is conducted, which complementarily makes the best of two modalities under different noisy conditions. Adaptive weights for decision fusion are generated using a sigmoid function based on reliabilities of the two modalities, capable of adapting to various noisy conditions. Experiments show that our proposed parallel AVKWS strategy based on decision fusion significantly improves the noise robustness and attains better performance than feature fusion based audio-visual spotter. Additionally, disCLBP-TOP shows more competitive performance than CLBP-TOP.

show abstract

Section: Adaptive Decision Fusionmentioning

confidence: 99%

Audio-visual Keyword Spotting for Mandarin Based on Discriminative Local Spatial-Temporal Descriptors

Liu¹,

Fan²,

Wu³

2014

2014 22nd International Conference on Pattern Recognition

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is also the case for works about multimodal intention recognition in other contexts than elderly assistance [12]. Some specialized approaches for audio-visual speech recognition [20], [21] considered uncertainty by performing uncertainty-based weighting for the fusion of multiple classifiers' outputs. In these works, the respective two categorical probability distributions returned by two individual classifiers for audio and visual input were combined by a weighted sum.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

Trick

Koert

Peters

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Assistive robots can potentially improve the quality of life and personal independence of elderly people by supporting everyday life activities. To guarantee a safe and intuitive interaction between human and robot, human intentions need to be recognized automatically. As humans communicate their intentions multimodally, the use of multiple modalities for intention recognition may not just increase the robustness against failure of individual modalities but especially reduce the uncertainty about the intention to be predicted. This is desirable as particularly in direct interaction between robots and potentially vulnerable humans a minimal uncertainty about the situation as well as knowledge about this actual uncertainty is necessary. Thus, in contrast to existing methods, in this work a new approach for multimodal intention recognition is introduced that focuses on uncertainty reduction through classifier fusion. For the four considered modalities speech, gestures, gaze directions and scene objects individual intention classifiers are trained, of which all output a probability distribution over all possible intentions. By combining these output distributions using the Bayesian method Independent Opinion Pool [1] the uncertainty about the intention to be predicted can be decreased. The approach is evaluated in a collaborative human-robot interaction task with a 7-DoF robot arm. The results show that fused classifiers which combine multiple modalities outperform the respective individual base classifiers with respect to increased accuracy, robustness, and reduced uncertainty.

show abstract

“…The input speech and lip movement video are recognized as a k* class using: For decision level fusion, the input speech and the lipmovement features are used to separately train an HMM classifier. The input speech and lip movement are classified together as c* class using [4]:…”

Section: Our Proposed Audio-visualmentioning

confidence: 99%

“…Liu et.al. proposed the mechanism of decision level fusion between the audio and visual features, and introduced a weighting scheme between each modality by means of reliability measures [4]. All of this research focuses on non-Thai commands.…”

Section: Introductionmentioning

confidence: 99%

Integrating lip-reading and thai speech to control electronic devices in a vehicle

Masamae

Chaikan

2015

2015 5th IEEE International Conference on System Engineering and Technology (ICSET)

View full text Add to dashboard Cite

This paper presents the use of lip-reading and Thai speech to control electronic devices in a vehicle. The Viola-Jones algorithm detects the face of the driver and the constrained local model detects their mouth area before three lips features are extracted. Hidden Markov models are utilized to recognize speech and lip movement, with the lip movement recognizer offering better accuracy than the speech recognizer in a noisy environment. Three fusion methods are utilized to combine lipmovement and speech. We propose the use of vehicle speed for selecting the appropriate recognizer for different speech signalto-noise ratios.

show abstract

Audio-visual keyword spotting based on adaptive decision fusion under noisy conditions for human-robot interaction

Cited by 13 publications

References 13 publications

Audio-visual Keyword Spotting for Mandarin Based on Discriminative Local Spatial-Temporal Descriptors

Audio-visual Keyword Spotting for Mandarin Based on Discriminative Local Spatial-Temporal Descriptors

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

Integrating lip-reading and thai speech to control electronic devices in a vehicle

Contact Info

Product

Resources

About