Lip Reading in Cantonese

Abstract

Lip reading aims at recognizing texts from a talking face without audio information. Due to the rapid development of deep learning techniques, researchers have made giant breakthroughs for both word-level and sentence-level English lip reading in recent years. Unlike English, it is difficult for Chinese to distinguish the lexical meanings, because Chinese is a tonal language. In addition, most of the existing Chinese lip reading datasets are designed for Mandarin, there are few for Cantonese. In this paper, we propose a word-level Cantonese lip reading dataset called CLRW which contains 800-word classes with 400,000 samples. For better practical applications, we do not limit gender, age, postures, light conditions, and speech speed to make CLRW closer to the real scene distribution. At first, we give a detailed description of the data collection process. Next, a novel two-branch network is proposed by us, named TBGL, which consists of a global branch and a local branch. The global branch models the whole lip and the local branch divides the feature into three parts to focus on subtle local lip motion. We jointly train these two branches and achieve comparable performance on LRW, CAS-VSR-W1K, and CLRW, respectively. Finally, we benchmark our dataset and perform a comprehensively analyze of the results, which demonstrate that CLRW is full of challenge, and it will bring a positive impact on further Cantonese lip reading tasks.

Cantonese sentence dataset for lip‐reading

“…Comprising 50 frequently used Tibetan words, the dataset initially featured 20 native speakers, equally divided by gender, and was later augmented using data enhancement techniques to produce 36,000 videos, averaging 720 videos per sample. Recently, Teng [8] propose a word-level Cantonese lip-reading dataset called CLRW which contains 800word classes with 400,000 samples. Dai et al [29] introduced a new Cantonese in-car audio-visual speech recognition (CI-AVSR) dataset for in-car command recognition in Cantonese.…”

“…The CLRS, CI-AVSR [29], and CLRW [8] datasets are all significant resources in the field of lipreading. The CLRS dataset is a multimodal corpus tailored for Cantonese sentence-level lipreading, comprising over 30,000 natural Cantonese sentences and recordings from more than 1,000 speakers.…”

“…The linguistic uniqueness of Cantonese, along with its extensive speakership, highlights the importance of investigating Cantonese lip-reading. In recent work, Teng et al [8] developed the first large-scale Cantonese word-level lip-reading dataset, Cantonese lip reading in the wild (CLRW). This dataset encompasses various factors such as age, gender, identity, program format, lighting, and facial resolution, containing 800 word classes and 400,000 word samples that are both challenging and realistic.…”

Application of convolutional neural networks to spoken words evaluation based on lip movements without accompanying sound signal

“…Several approaches related to lip reading are briefly addressed in this paper. The authors [3] presented the method of detecting lips and using the cropped images as a dataset for the training set for Convolutional Neural Networks. Also, they discussed different methods of evaluation that can be used.…”