Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition

Wang, Jianrong; Tang, Ziyue; Li, Xuewei; Yu, Mo; Fang, Qiang; Liu, Li

doi:10.21437/interspeech.2021-432

Cited by 10 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For these particular experimental conditions, our proposed method outperforms the one reported in [12] (77.1% vs. 70.9% [12]). However, when considering the original CSF corpus, our system performs significantly worse than the one proposed in [13] (64.9% vs. 74.2%) which is based on a more complex architecture (pre-trained teacher model with knowledge distillation toward a student model). When considering the corrected CSF18V2 corpus, our method reaches a comparable level of performance (70.9%).…”

Section: Resultsmentioning

confidence: 74%

“…We now compare the different architectures used to com- For all experiments, ∆ 95% confidence interval is around 4%. The results for the other approaches are reported as is from the papers [6], [12], [13].…”

Section: Resultsmentioning

confidence: 99%

“…When considering the corrected CSF18V2 corpus, our method reaches a comparable level of performance (70.9%). Thus, it would be interesting for future work to quantify the extra benefit given by knowledge distillation approach of [13] on this updated dataset.…”

Section: Resultsmentioning

confidence: 99%

“…In [12], Papatimitriou et al proposed a fully convolutional model with a time-depth separable block and attention based decoder. In [13], Wang et al make use of pre-trained teacher model by exploring different methods for cross-modal knowledge distillation. Both approaches are evaluated on the CSF18 corpus and outperformed the CNN-HMM baseline by a significant margin.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding

Sankar¹,

Beautemps²,

Hueber³

2022

Preprint

View full text Add to dashboard Cite

This paper proposes a simple and effective approach for automatic recognition of Cued Speech (CS), a visual communication tool that helps people with hearing impairment to understand spoken language with the help of hand gestures that can uniquely identify the uttered phonemes in complement to lip-reading. The proposed approach is based on a pre-trained hand and lips tracker used for visual feature extraction and a phonetic decoder based on a multistream recurrent neural network trained with connectionist temporal classification loss and combined with a pronunciation lexicon. The proposed system is evaluated on an updated version of the French CS dataset CSF18 for which the phonetic transcription has been manually checked and corrected. With a decoding accuracy at the phonetic level of 70.88%, the proposed system outperforms our previous CNN-HMM decoder and competes with more complex baselines.

show abstract

Section: Resultsmentioning

confidence: 74%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding

Sankar¹,

Beautemps²,

Hueber³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…More precisely, the hand shapes are used to code consonants, while the hand positions on one side of the face or the neck are used to code vowels. Nowadays, it is estimated that CS has been adapted to over sixty languages, such as English CS [8,9,10,11,12], and French CS hand slides are used, we propose an novel alternative Mandarin Chinese CS system called MCCS-2 (see Fig. 1) using hand slides to code diphthongs.…”

Section: Introductionmentioning

confidence: 99%

Objective Hand Complexity Comparison between Two Mandarin Chinese Cued Speech Systems

Liu

Feng

Ren

et al. 2022

2022 13th International Symposium on Chinese Spoken Language Processing (ISCSLP)

Self Cite

View full text Add to dashboard Cite

Figure 1: Comparsion of vowel allocations between the proposed MCCS-2 and MCCS-1 in the previous work [16].[ 13,14,15].Recently, a new Mandarin Chinese CS system (MCCS) was proposed for Mandarin Chinese [16], which is called MCCS-1 in the current paper. In this system, a key approach that all the compound finals starting with i [i], u [u], ü [y] are coded by semi-consonants [j], [w], [4] was proposed to reduce the number of vowels from 35 to 15. This is because the compound finals are coded by hand shapes instead of hand positions. Then each of remained 15 vowels is allocated to one specific hand position, without using any hand slides 1 to code diphthongs or compound finals. This system satisfies two main criteria [17] that: 1) hand and lips coding should be complementary (i.e., the phonemes which have similar lip shape should be distinguished by different hand shapes or hand positions); 2) cuers should spend as less as possible energy to code.To the best of our knowledge, there are no research works concerning the CS system optimization based on the cuer's hand kinematics analysis. However, some works are dedicated to the kinematics and kinetics of hand or arm movements in other related domains. For example, it was studied the hand movement towards a target in [18,19,20], and the kinematic features of the arm movements were studied in [21,22,23]. Particularly, in [24], Grujic and Bonkovic described measurement and analysis of human hand kinematics, and [25] analyzed kinematic parameters for handwriting. Moreover, Nelson [26] described in detail physical principles for economics of skilled movements, in which the jaw movements during speech were analyzed. Inspired by these studies, we adopted three fundamental kinematic parameters, i.e. time duration, motion trajectory length, tangential speed to the study of the CS cuer's hand movement.In order to demonstrate that MCCS-1 system which does not use hand slides, possess advantages over systems in which 1 Hand moves from one position to another.

show abstract