Lip contour segmentation and tracking compliant with lip-reading application constraints

Stillittano, Sébastien; Girondel, Vincent; Caplier, Alice

doi:10.1007/s00138-012-0445-1

Cited by 23 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early studies of ACSR tended to directly concatenate the features of multi-modal inputs as the dominant fusion paradigm. For example, [41], [42] used different colors to mark lip and hand regions for further feature extraction and fusion, as well as the coordinates of the marks on the finger. The regions of interest (ROIs) were segmented to extract the ROI-based features of lips and hands, which exploited a pre-defined threshold to track the marks of cuers [6], [43].…”

Section: A Multi-modal Fusion In Acsrmentioning

confidence: 99%

A shutter pulse generator for high speed camera

Liu¹,

Li²,

Li³

2013

JOURNAL OF SHENZHEN UNIVERSITY SCIENCE AND ENGINEERING

View full text Add to dashboard Cite

Cued Speech (CS) is a pure visual coding method used by hearing-impaired people that combines lip reading with several specific hand shapes to make the spoken language visible. Automatic CS recognition (ACSR) seeks to transcribe visual cues of speech into text, which can help hearing-impaired people to communicate effectively. The visual information of CS contains lip reading and hand cueing, thus the fusion of them plays an important role in ACSR. However, most previous fusion methods struggle to capture the global dependency present in long sequence inputs of multi-modal CS data. As a result, these methods generally fail to learn the effective cross-modal relationships that contribute to the fusion. Recently, attentionbased transformers have been a prevalent idea for capturing the global dependency over the long sequence in multi-modal fusion, but existing multi-modal fusion transformers suffer from both poor recognition accuracy and inefficient computation for the ACSR task. To address these problems, we develop a novel computation and parameter efficient multi-modal fusion transformer by proposing a novel Token-Importance-Aware Attention mechanism (TIAA), where a token utilization rate (TUR) is formulated to select the important tokens from the multi-modal streams. More precisely, TIAA firstly models the modality-specific fine-grained temporal dependencies over all tokens of each modality, and then learns the efficient cross-modal interaction for the modality-shared coarse-grained temporal dependencies over the important tokens of different modalities. Besides, a lightweight gated hidden projection is designed to control the feature flows of TIAA. The resulting model, named Economical Cued Speech Fusion Transformer (EcoCued), achieves state-of-the-art performance on all existing CS datasets (i.e., Mandarin Chinese, French, and British CS), compared with existing transformerbased fusion methods and ACSR fusion methods. Notably, our method dramatically reduces the computational complexity from O(T 2 ) to O(T ). We will release the source code and data as open source.

show abstract

Section: A Multi-modal Fusion In Acsrmentioning

confidence: 99%

A shutter pulse generator for high speed camera

Liu¹,

Li²,

Li³

2013

JOURNAL OF SHENZHEN UNIVERSITY SCIENCE AND ENGINEERING

View full text Add to dashboard Cite

show abstract

“…Large training set and good initial condition are necessary for model-based technique. Stillitano et al [4] proposed to use both active contours and parametric models for lip contour extraction. This method need prior knowledge of the lip shape.…”

Section: Introductionmentioning

confidence: 99%

Automatic dynamic template tracking of inner lips based on CLNF

Liu

Feng

Beautemps

2017

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this paper, a novel automatic approach to extract the inner lips contour of speakers without using artifices is proposed. This method is based on a recent facial contour extraction model developed in computer vision, called Constrained Local Neural Field (CLNF), which provides 8 characteristic points (landmarks) defining the inner lips contour. However, directly applied to our visual data including Cued Speech (CS) data, CLNF failed in about 50% of cases. We propose a Modified CLNF to estimate inner lips contour based on original CLNF landmarks. A dynamic template using the first derivative of smoothed luminance variation is explored in this new model. This method gives precise estimation of aperture for inner lips. It is evaluated on 4800 images of three French speakers. The proposed method corrects 95% CLNF errors and total RMSE of one pixel (i.e. 0.05cm in average) is reached, instead of four pixels using original CLNF.

show abstract

“…Some authors used parametric models in which the lips shape is described by curves controlled by a limited set of parameters. Stillitano et al [11] used both active contours and parametric models for lips contour extraction. This method needs prior knowledge of the lips shape.…”

Section: Introductionmentioning

confidence: 99%

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Liu¹,

Feng²,

Beautemps³

2017

The 14th International Conference on Auditory-Visual Speech Processing

View full text Add to dashboard Cite

In this paper, a novel automatic method using an adaptive ellipse model to estimate inner lips parameters (inner lips width A and height B) of speakers without any artifices is presented. Color based image processing is first applied to segment preliminary inner lips. A single discontinuity elimination combining horizontal and vertical filling are used to obtain a binary inner lips image as complete as possible. After the previous pre-processing steps, an optimal adaptive ellipse is determined to match the inner lips. The proposed method is evaluated on 4693 images of three French speakers including one Cued Speech (CS) speaker. It obtains RMSE of 3.37 mm for A parameter and of 0.84 mm for B parameter which outperform the baseline of inner lips parameter estimation in the state of the art. Moreover, CS recognition based on 34 French phonemes shows that using the estimated two parameters achieves an accuracy which is comparable to that using raw lips ROI.

show abstract

Lip contour segmentation and tracking compliant with lip-reading application constraints

Cited by 23 publications

References 19 publications

A shutter pulse generator for high speed camera

A shutter pulse generator for high speed camera

Automatic dynamic template tracking of inner lips based on CLNF

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Contact Info

Product

Resources

About