Automatic dynamic template tracking of inner lips based on CLNF

Liu, Li; Feng, Gang; Beautemps, Denis

doi:10.1109/icassp.2017.7953134

Cited by 6 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In summary, our results show a far superior performance of A and B parameter estimation to that in the state of the art, and also are comparable to our previous work [18] which only addressed B parameter estimation. To further evaluate the performance of the proposed method, we applied the estimated A and B parameters of inner lips to CS recognition on 34 French phonemes.…”

Section: Evaluation and Resultssupporting

confidence: 84%

“…In the previous CS study of lips parameter estimation, Heracleous et al [1] extracted the lips A and B parameters by painting blue color on subject's lips and tracking the blue lips. A dynamic correlation template method based on Constrained Local Neural fields (CLNF) [16] in our previous work (without any artifice color marks) [17,18] is investigated to estimate B parameter. CLNF is proposed by Baltusaitis et al in 2013, which is robust for facial landmark detection in the wild.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Liu¹,

Feng²,

Beautemps³

2017

The 14th International Conference on Auditory-Visual Speech Processing

Self Cite

View full text Add to dashboard Cite

In this paper, a novel automatic method using an adaptive ellipse model to estimate inner lips parameters (inner lips width A and height B) of speakers without any artifices is presented. Color based image processing is first applied to segment preliminary inner lips. A single discontinuity elimination combining horizontal and vertical filling are used to obtain a binary inner lips image as complete as possible. After the previous pre-processing steps, an optimal adaptive ellipse is determined to match the inner lips. The proposed method is evaluated on 4693 images of three French speakers including one Cued Speech (CS) speaker. It obtains RMSE of 3.37 mm for A parameter and of 0.84 mm for B parameter which outperform the baseline of inner lips parameter estimation in the state of the art. Moreover, CS recognition based on 34 French phonemes shows that using the estimated two parameters achieves an accuracy which is comparable to that using raw lips ROI.

show abstract

Section: Evaluation and Resultssupporting

confidence: 84%

Section: Introductionmentioning

confidence: 99%

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Liu¹,

Feng²,

Beautemps³

2017

The 14th International Conference on Auditory-Visual Speech Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…In our recent works [25], [26], [27], several methods to get rid of these artifices on the speaker's lips and hand were explored. A modified constrained local neural fields (CLNF) model was proposed to extract the inner lips height and width, and an adaptive ellipse model was proposed for inner lips parameters extraction in CS.…”

Section: A Feature Extraction In Csmentioning

confidence: 99%

Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

Liu

Feng

Beautemps

et al. 2021

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Cued Speech (CS) is an augmented lip reading system complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6% has been achieved retaining 76.6% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04%), which did not take into account the asynchrony issue of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.

show abstract

“…In this paper, we improve this method for correcting the B parameter by a HD-CTM, which is robust for tracking any lips shape. More precisely, compared with [14], three improvements are proposed to enhance the robustness of our method. They contain a new initial condition for the first image, an automatic tracking of the starting position for template searching region, and a novel dynamic hybrid template.…”

Section: Relation With Prior Workmentioning

confidence: 99%

“…Then, a threshold was applied to the resulted image to segment the blue lips. To get rid of the blue lips, a dynamic correlation template method in our previous work [14] was introduced to estimate B parameters of inner lips. In this paper, we improve this method for correcting the B parameter by a HD-CTM, which is robust for tracking any lips shape.…”

Section: Relation With Prior Workmentioning

confidence: 99%

Inner lips feature extraction based on CLNF with hybrid dynamic template for Cued Speech

Liu

Feng

Beautemps

2017

J Image Video Proc.

Self Cite

View full text Add to dashboard Cite

In previous French Cued Speech (CS) studies, one of the widely used methods is painting blue color on the speaker's lips to make lips feature extraction easier. In this paper, in order to get rid of this artifice, a novel automatic method to extract the inner lips contour of CS speakers is presented. This method is based on a recent facial contour extraction model developed in computer vision, called Constrained Local Neural Field (CLNF), which provides eight characteristic landmarks describing the inner lips contour. However, directly applied to our CS data, CLNF fails in about 41.4% of cases. Therefore, we propose two methods to correct the B parameter (aperture of inner lips) and A parameter (width of inner lips), respectively. For correcting the B parameter, a hybrid dynamic correlation template method (HD-CTM) using the first derivative of smoothed luminance variation is proposed. HD-CTM is first applied to detect the outer lower lips position. Then, the inner lower lips position is obtained by subtracting the validated lower lips thickness (VLLT). For correcting the A parameter, a periodical spline interpolation with a geometrical deformation of six CLNF inner lips landmarks is explored. Combined with an automatic round lips detector, this method is efficient to correct A parameter for round lips (the third vowel viseme made of French vowels with a small opening). HD-CTM is evaluated on 4800 images of three French speakers. It corrects about 95% CLNF errors of the B parameter, and total RMSE of one pixel (i.e., 0.05 cm on average) is achieved. The periodical spline interpolation method is tested on 927 round lips images. The total error of CLNF is reduced significantly, which is comparable to the state of the art. Moreover, the third viseme is properly distributed in the parameter A and B plane after using this method.

show abstract

Automatic dynamic template tracking of inner lips based on CLNF

Cited by 6 publications

References 8 publications

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Inner Lips Parameter Estimation based on Adaptive Ellipse Model

Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

Inner lips feature extraction based on CLNF with hybrid dynamic template for Cued Speech

Contact Info

Product

Resources

About