Integrating auditory and visual representations for audiovisual vowel recognition

Robert-Ribes, Jordi; Lallouache, Tahar; Escudier, Pierre; Schwartz, Jean‐Luc

doi:10.21437/eurospeech.1993-405

Cited by 4 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several methods for integration of audio and visual sources have been proposed (e.g. [5,4,6,9,10,11]). Robert-Ribes [4] has proposed a classication scheme for integra-tion strategies.…”

Section: Integration Of Audio and Visual Informationmentioning

confidence: 99%

Robust audiovisual integration using semicontinuous hidden Markov models

Su¹,

Silsbee²

1996

4th International Conference on Spoken Language Processing (ICSLP 1996)

View full text Add to dashboard Cite

We d escribe an improved method of integrating a udio and visual information in a HMM-based audiovisual ASR system. The m ethod uses a modied semicontinuous HMM (SCHMM) for integration and recognition. Our results show substantial improvements over earlier integration methods at high noise levels.Our integration method relies on the assumption that, as environmental conditions deviate from those under which training occurred, the u n d erlying probability distributions will also change. We u s e p h oneme based SCHMMs for classication of isolated words. The probability m o d els underlying t h e standard SCHMM are Gaussian; thus, low probability e s t imates will tend t o be associated with high condences (small dierences in the feature values cause large proportional differences in probabilities, when the v alues are in the t ail of the distribution). Therefore, during classication, we replace each G a ussian with a scoring f u nction which looks Gaussian near the m ean of the distribution but h as a heavier tail.We report results comparing t his method with a n a udioonly system and with previous integration methods. At high noise levels, the system with modied scoring f u nctions shows a b e t t er than 50recognition does suer when noise is low. Methods which can adjust the relative w eight o f t h e a udio and visual information can still potentially outperform the new method, provided that a reliable way of choosing t h ose weights can be found.

show abstract

“…Several methods for integration of audio and visual sources have been proposed (e.g. [5,4,6,9,10,11]). Robert-Ribes [4] has proposed a classication scheme for integra-tion strategies.…”

Section: Integration Of Audio and Visual Informationmentioning

confidence: 99%

Robust audiovisual integration using semicontinuous hidden Markov models

Su¹,

Silsbee²

1996

4th International Conference on Spoken Language Processing (ICSLP 1996)

View full text Add to dashboard Cite

show abstract

A comparison of models for fusion of the auditory and visual sensors in speech perception

1995

View full text Add to dashboard Cite

Robust audiovisual integration using semicontinuous hidden Markov models

Qin

Silsbee²

Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96

View full text Add to dashboard Cite

We d escribe an improved method of integrating a udio and visual information in a HMM-based audiovisual ASR system. The m ethod uses a modied semicontinuous HMM (SCHMM) for integration and recognition. Our results show substantial improvements over earlier integration methods at high noise levels. Our integration method relies on the assumption that, as environmental conditions deviate from those under which training occurred, the u n d erlying probability distributions will also change. We u s e p h oneme based SCHMMs for classication of isolated words. The probability m o d els underlying t h e standard SCHMM are Gaussian; thus, low probability e s t imates will tend t o be associated with high condences (small dierences in the feature values cause large proportional differences in probabilities, when the v alues are in the t ail of the distribution). Therefore, during classication, we replace each G a ussian with a scoring f u nction which looks Gaussian near the m ean of the distribution but h as a heavier tail. We report results comparing t his method with a n a udioonly system and with previous integration methods. At high noise levels, the system with modied scoring f u nctions shows a b e t t er than 50recognition does suer when noise is low. Methods which can adjust the relative w eight o f t h e a udio and visual information can still potentially outperform the new method, provided that a reliable way of choosing t h ose weights can be found.

show abstract

Integrating auditory and visual representations for audiovisual vowel recognition

Cited by 4 publications

References 0 publications

Robust audiovisual integration using semicontinuous hidden Markov models

Robust audiovisual integration using semicontinuous hidden Markov models

A comparison of models for fusion of the auditory and visual sensors in speech perception

Robust audiovisual integration using semicontinuous hidden Markov models

Contact Info

Product

Resources

About