DBN based multi-stream models for audio-visual speech recognition

Gowdy, J.N.; Subramanya, Amarnag; Bartels, Chris; Bilmes, Jeff

doi:10.1109/icassp.2004.1326155

Cited by 53 publications

(35 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We establish that the correct way to treat asynchrony in audio-visual speech recognition is within word boundaries and propose a new DBN phoneme model able to exploit its asynchrony without being limited to small vocabulary tasks. Our model thus enjoys the benefits of the DBNs documented in [3], [4], [5] overcoming the problems encountered by Graviet et al when extending their use to phonemes and large vocabulary tasks [6].…”

Section: Introductionmentioning

confidence: 89%

“…Previous works [3], [4], [5] and our own experiments, see section IV, show the benefits of allowing audio-visual asynchrony within word boundaries. Analyzing why those asynchronous DBN models work better than the traditional MSHMMs, we develop a processing technique to overcome asynchrony by an additional processing step on visual features when MSHMMs are used for recognition.…”

Section: A Preliminary Idea Based On Model Analysismentioning

confidence: 99%

“…Figure 3 and 4 show the DBN representation of words with IHMMs and pHMMs. It is important to note that those models are not word IHMMs [3], [2], [4], whose grouped states correspond to phonemes. Such a system would be defining word-depending phoneme models, which can only be used in reduced vocabulary tasks.…”

Section: Proposed Asynchronous Modelmentioning

confidence: 99%

“…However, several works [3], [4], [5] have proved the benefits of the more general Dynamic Bayesian Network (DBN) models allowing asynchrony between the audio and the visual streams. In that case, DBNs need to define some synchronization points for the models.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Overcoming asynchrony in Audio-Visual Speech Recognition

Estellers

Thiran

2010

2010 IEEE International Workshop on Multimedia Signal Processing

View full text Add to dashboard Cite

Abstract-In this paper we propose two alternatives to overcome the natural asynchrony of modalities in Audio-Visual Speech Recognition. We first investigate the use of asynchronous statistical models based on Dynamic Bayesian Networks with different levels of asynchrony. We show that audio-visual models should consider asynchrony within word boundaries and not at phoneme level. The second approach to the problem includes an additional processing of the features before being used for recognition. The proposed technique aligns the temporal evolution of the audio and video streams in terms of a speechrecognition system and enables the use of simpler statistical models for classification. On both cases we report experiments with the CUAVE database, showing the improvements obtained with the proposed asynchronous model and feature processing technique compared to traditional systems.

show abstract

Section: Introductionmentioning

confidence: 89%

Section: A Preliminary Idea Based On Model Analysismentioning

confidence: 99%

Section: Proposed Asynchronous Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Overcoming asynchrony in Audio-Visual Speech Recognition

Estellers

Thiran

2010

2010 IEEE International Workshop on Multimedia Signal Processing

View full text Add to dashboard Cite

show abstract

“…Early multistream work also includes that of HMM decomposition [7], where both speech and noise are consider a separate stream. Dynamic Bayesian networks (DBNs) have also been used for multi-stream [8], including audio-visual speech recognition [9,10,11]. Even HTK has the ability to represent multiple synchronous acoustic streams.…”

Section: Introductionmentioning

confidence: 99%

Polyphase speech recognition

Lin

Bilmes

2008

2008 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

We propose a model for speech recognition that consists of multiple semi-synchronized recognizers operating on a polyphase decomposition of standard speech features. Specifically, we consider multiple out-of-phase downsampled speech features as separate streams which are modeled separately at the lowest level, and are then integrated at the higher level (words) during first-pass decoding. Our model lessens the severity of the oversampling problem in many speech recognition systems -i.e., that speech modulation energy is most important below 25Hz but a 100Hz frame rate gives a modulation bandwidth of 50Hz. Our polyphase approach moreover captures wider and more diverse dynamics within the speech signal. Our integrative network is high-level, namely it couples together and decodes word strings from different recognizers simultaneously and asynchronously. We provide preliminary results on the 10-word vocabulary version of the SVitchboard (small-vocabulary switchboard) task and show that our polyphase recognition system significantly outperforms an optimized baseline (HMM) approach.

show abstract

Multi-level Fusion of Audio and Visual Features for Speaker Identification

Cai

Meng

2005

Advances in Biometrics

View full text Add to dashboard Cite

This paper explores the fusion of audio and visual evidences through a multi-level hybrid fusion architecture based on dynamic Bayesian network (DBN), which combines model level and decision level fusion to achieve higher performance. In model level fusion, a new audiovisual correlative model (AVCM) based on DBN is proposed, which describes both the intercorrelations and loose timing synchronicity between the audio and video streams. The experiments on the CMU database and our own homegrown database both demonstrate that the methods can improve the accuracies of audiovisual bimodal speaker identification at all levels of acoustic signal-to-noiseratios (SNR) from 0dB to 30dB with varying acoustic conditions.

show abstract

DBN based multi-stream models for audio-visual speech recognition

Cited by 53 publications

References 6 publications

Overcoming asynchrony in Audio-Visual Speech Recognition

Overcoming asynchrony in Audio-Visual Speech Recognition

Polyphase speech recognition

Multi-level Fusion of Audio and Visual Features for Speaker Identification

Contact Info

Product

Resources

About