Lip reading using external viseme decoding

Peymanfard, Javad; Mohammadi, Mohammad Reza; Zeinali, Hossein; Mozayani, Nasser

doi:10.1109/mvip53647.2022.9738749

Cited by 6 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chung et al [15] extended the WAS model to the MV-WAS model that can decode visual sequences across all poses and show that it is possible to read lips in profile, but the standard is inferior to reading frontal faces. Peymanfard et al [29] suggested external viseme decoding, which divides the sequence-to-sequence model into two stages, video to viseme and viseme to character, respectively. The network outputs character sequence given viseme through external text data.…”

Section: Related Workmentioning

confidence: 99%

“…In this step, we expected the phonemes having the same visual features, like the same lip movements and forms, to be grouped in the same category. We used the method in [29] to train our model. Employing the two models of the video-to-viseme and the viseme-to-character models and finally merging the two, we obtained the lip reading model.…”

Section: Viseme Analysismentioning

confidence: 99%

See 1 more Smart Citation

A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset

Peymanfard¹,

Heydarian²,

Lashini³

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, significant progress has been made in automatic lip reading. But these methods require large-scale datasets that do not exist for many low-resource languages. In this paper, we have presented a new multipurpose audio-visual dataset for Persian. This dataset consists of almost 220 hours of videos with 1760 corresponding speakers. In addition to lip reading, the dataset is suitable for automatic speech recognition, audio-visual speech recognition, and speaker recognition. Also, it is the first large-scale lip reading dataset in Persian. A baseline method was provided for each mentioned task. In addition, we have proposed a technique to detect visemes (a visual equivalent of a phoneme) in Persian. The visemes obtained by this method increase the accuracy of the lip reading task by 7% relatively compared to the previously proposed visemes, which can be applied to other languages as well.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Viseme Analysismentioning

confidence: 99%