Audiovisual Sensory Integration Using Hidden Markov Models

Silsbee, Peter L.; Qin, Su

doi:10.1007/978-3-662-13015-5_37

Cited by 9 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been various implementations of the DI architecture in speech recognition (see, e.g. [2], [12], [18], [21], [30], [42], [44], [49], [54], and [59]) and psychophysical modeling (see [11] and 16]). …”

Section: A Theoretical Backgroundmentioning

confidence: 99%

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Teissier

Robert-Ribes

Schwartz

et al. 1999

IEEE Trans. Speech Audio Process.

View full text Add to dashboard Cite

Audiovisual speech recognition involves fusion of the audio and video sensors for phonetic identification. There are three basic ways to fuse data streams for taking a decision such as phoneme identification: data-to-decision, decision-to-decision, and data-to-data. This leads to four possible models for audiovisual speech recognition, that is direct identification in the first case, separate identification in the second one, and two variants of the third early integration case, namely dominant recoding or motor recoding. However, no systematic comparison of these models is available in the literature. We propose an implementation of these four models, and submit them to a benchmark test. For this aim, we use a noisy-vowel corpus tested on two recognition paradigms in which the systems are tested at noise levels higher than those used for learning. In one of these paradigms, the signal-to-noise ratio (SNR) value is provided to the recognition systems, in the other it is not. We also introduce a new criterion for evaluating performances, based on transmitted information on individual phonetic features.In light of the compared performances of the four models with the two recognition paradigms, we discuss the advantages and drawbacks of these models, leading to proposals for data representation, fusion architecture, and control of the fusion process through sensor reliability.

show abstract

Section: A Theoretical Backgroundmentioning

confidence: 99%

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Teissier

Robert-Ribes

Schwartz

et al. 1999

IEEE Trans. Speech Audio Process.

View full text Add to dashboard Cite

show abstract

“…Using combined audio and visual features, recognition performance was improved by a maximum of 10% at high and low SNR's over an audio-only recogniser. Future work will focus on finding more effective ways of combining the audio and visual information with the aim of ensuring that the combined performance is always at least as good as the performance using either modality [1,14,16,17] and in deriving more discriminative features from the scale histogram.…”

Section: Discussionmentioning

confidence: 99%

“…It has already been shown [1,6,8,10,13,15,16,17] that the incorporation of visual information with acoustic speech recognition leads to a more robust recogniser. While the visual cues of speech alone are unable to discriminate between all phonemes (e.g.…”

Section: Introductionmentioning

confidence: 99%

Audiovisual speech recognition using multiscale nonlinear image decomposition

Matthews

Bangham²,

Cox³

Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96

View full text Add to dashboard Cite

There has recently been increasing interest in the idea of enhancing speech recognition by the use of visual information derived from the face of the talker. This paper demonstrates the use of nonlinear image decomposition, in the form of a 'sieve', applied to the task of visual speech recognition. Information derived from the mouth region is used in visual and audiovisual speech recognition of a database of the letters A-Z for four talkers. A scale histogram is generated directly from the grayscale pixels of a window containing the talkers mouth on a per frame basis. Results are presented for visual-only, audio-only and in a simple audiovisual case.

show abstract

“…The models in this study were trained only with 'clean' data and used a single system. Other approaches using HMMs have attempted to compute an explicit weighting factor to bias the recognition in favour of the visual compoinent when acoustic noise levels increase and thus to build an adaptive system [1,17]. Although all of the systems have demonstrated the benefits of adding a visual component to the recognition process, especially when the acoustic noise levels are high, none has yet clearly demonstrated a bimodal performance that is uniformly better than the unimodal performance in either of the two domains.…”

Section: Architectures Using Hmmsmentioning

confidence: 99%

Using the visual component in automatic speech recognition

Brooke

Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96

View full text Add to dashboard Cite

The movements of talkers' faces are known to convey visual cues that can improve speech intelligibility, especially where there is noise or hearing-impairment. This suggests that visible facial gestures could be exploited to enhance speech intelligibility in automatic systems. Handling the volume of data represented by images of talkers' faces implies some form of data compression. Rather than using conventional feature extraction approaches, image coding and compression can be achieved using datadriven, statistically-oriented techniques such as artificial neuralnetworks (ANNs) or principal component analysis (PCA). A major issue is the combination of the audio and visual data so that the best use can be made of the two modalities together. Perceptual experiments may offer guidance on suitable machine architectures, many of which currently use hidden Markov models (HMMs).

show abstract

Audiovisual Sensory Integration Using Hidden Markov Models

Cited by 9 publications

References 0 publications

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Audiovisual speech recognition using multiscale nonlinear image decomposition

Using the visual component in automatic speech recognition

Contact Info

Product

Resources

About