Robust audiovisual integration using semicontinuous hidden Markov models

Abstract

We d escribe an improved method of integrating a udio and visual information in a HMM-based audiovisual ASR system. The m ethod uses a modied semicontinuous HMM (SCHMM) for integration and recognition. Our results show substantial improvements over earlier integration methods at high noise levels. Our integration method relies on the assumption that, as environmental conditions deviate from those under which training occurred, the u n d erlying probability distributions will also change. We u s e p h oneme based SCHMMs for classication of isolated words. The probability m o d els underlying t h e standard SCHMM are Gaussian; thus, low probability e s t imates will tend t o be associated with high condences (small dierences in the feature values cause large proportional differences in probabilities, when the v alues are in the t ail of the distribution). Therefore, during classication, we replace each G a ussian with a scoring f u nction which looks Gaussian near the m ean of the distribution but h as a heavier tail. We report results comparing t his method with a n a udioonly system and with previous integration methods. At high noise levels, the system with modied scoring f u nctions shows a b e t t er than 50recognition does suer when noise is low. Methods which can adjust the relative w eight o f t h e a udio and visual information can still potentially outperform the new method, provided that a reliable way of choosing t h ose weights can be found.