Finding Lips in Unconstrained Imagery for Improved Automatic Speech Recognition

Abstract

Lip movement of a speaker conveys important visual speech information and can be exploited for Automatic Speech Recognition. While previous research demonstrated that visual modality is a viable tool for identifying speech, the visual information has yet to become utilized in mainstream ASR systems. One obstacle is the difficulty in building a robust visual front end that tracks lips accurately in a real-world condition. In this paper we present our current progress in addressing the issue. We examine the use of color information in detecting the lip region and report our results on the statistical analysis and modeling of lip hue images by examining hundreds of manually extracted lip images obtained from several databases. In addition to hue color, we also explore spatial and edge information derived from intensity and saturation images to improve the robustness of the lip detection. Successful application of this algorithm is demonstrated over imagery collected in visually challenging environments.

Face and lip tracking in unconstrained imagery for improved automatic speech recognition

“…However, if a subject moves in a direction normal to the camera plane (closer or further to the camera) where the current candidate ROI will either no longer contain the full face or will contain additional background information, an ROI scaling algorithm was developed. It was found in previous research [21] that the expected value for the skin class in the shifted-hue (sH) color plane is lower than the background class. Therefore, if we determine the gradient of the candidate ROI for the sH color plane, then the gradient magnitudes are expected to be larger around the face perimeter.…”

“…To simplify processing within the hue component a 0.2 shift was then applied, resulting in the modified color space, sHSI. § © More details on color space analysis can be found in our previous work [21].…”

Building a Visual Front-end for Audio-Visual Automatic Speech Recognition in Vehicle Environments

“…To determine the optimal color space for efficient skin and face detection, various color spaces have been examined, such as RGB, nrgb, YcbCr, YIQ, and HSV in [11]. Manaully drawn lip masks were constructed from a database of over 400 images that were subsequently used to develop statistical models of Lip, Non-lip, and Skin classes.…”