A Survey of Lipreading Methods Based on Deep Learning

Hao, Mingfeng; Mamut, Mutelep; Ubul, Kurban

doi:10.1145/3421558.3421563

Cited by 5 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pioneering end-to-end CNN method DIH (Tsai et al 2017) encoded the input image and foreground mask, which was then decoded to the harmonized image and scene parsing image. Based on the encoderdecoder framework, several methods (Cun and Pun 2020;Hao et al 2020;Cong et al 2021) applied the attention mechanism to learn foreground and background appearance feature separately for harmonization. Moreover, RainNet (Ling et al 2021) designed a region-aware adaptive instance normalization module to transfer the visual style from background to foreground.…”

Section: Encoder-decoder Harmonization Methodsmentioning

confidence: 99%

FRIH: Fine-Grained Region-Aware Image Harmonization

Peng,

Luo,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

Image harmonization aims to generate a more realistic appearance of foreground and background for a composite image. All the existing methods perform the same harmonization process for the whole foreground. However, the implanted foreground always contains different appearance patterns. Existing solutions ignore the difference of each color block and lose some specific details. Therefore, we propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH). In the first stage, the whole input foreground mask is used to make a global coarse-grained harmonization. In the second stage, we adaptively cluster the input foreground mask into several submasks. Each submask and the coarsely adjusted image are concatenated respectively and fed into a lightweight cascaded module, refining the global harmonization result. Moreover, we further design a fusion prediction module to generate the final result, utilizing the different degrees of harmonization results comprehensively. Without bells and whistles, our FRIH achieves a competitive performance on iHarmony4 dataset with a lightweight model.

show abstract

Section: Encoder-decoder Harmonization Methodsmentioning

confidence: 99%

FRIH: Fine-Grained Region-Aware Image Harmonization

Peng,

Luo,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Firstly, We compare our portrait-background image harmonization method with five state-of-the-art image harmonization methods on the synthesized images. The compared methods include ADFM [HIF20] , HT [GGZ*21], iDIH-HRNet [SPK21], S2CRNet [LCPW21], and DoveNet [CZN*20]. We not only utilize their provided pretrained models for inference but also retrain their models on the synthesis dataset generated as proposed in our work.…”

Section: Comparison With Existing Methodsmentioning

confidence: 99%

“…The different cap-turing conditions and devices of the two images are the major reason causing the inconsistency. To eliminate the visual inconsistency, plenty of deep-learning-based methods have been proposed to harmonize the composited images [GGZ*21,JZZ*21,LXS*21,CNZ*21,CZN*20,SPK21,HIF20,CP20,TSL*17]. The image harmonization methods adjust foreground objects to make them compatible with backgrounds.…”

Section: Introductionmentioning

confidence: 99%

Harmonized Portrait‐Background Image Composition

Wang

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Portrait‐background image composition is a widely used operation in selfie editing, video meeting, and other portrait applications. To guarantee the realism of the composited images, the appearance of the foreground portraits needs to be adjusted to fit the new background images. Existing image harmonization approaches are proposed to handle general foreground objects, thus lack the special ability to adjust portrait foregrounds. In this paper, we present a novel end‐to‐end network architecture to learn both the content features and style features for portrait‐background composition. The method adjusts the appearance of portraits to make them compatible with backgrounds, while the generation of the composited images satisfies the prior of a style‐based generator. We also propose a pipeline to generate high‐quality and high‐variety synthesized image datasets for training and evaluation. The proposed method outperforms other state‐of‐the‐art methods both on the synthesized dataset and the real composited images and shows robust performance in video applications.

show abstract

“…[9] provided a review of traditional and deep learning based architectures grouped by tasks and datasets. Particularly, there are three works concentrating on deep learning in VSR: [10], [11], and [12]. These works mainly focus on the comparison of various methods and their performance and VSR datasets.…”

Section: Introductionmentioning

confidence: 99%

Advances and Challenges in Deep Lip Reading

Oghbaie¹,

Sabaghi²,

Kooshan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Driven by deep learning techniques and large-scale datasets, recent years have witnessed a paradigm shift in automatic lip reading. While the main thrust of Visual Speech Recognition (VSR) was improving accuracy of Audio Speech Recognition systems, other potential applications, such as biometric identification, and the promised gains of VSR systems, have motivated extensive efforts on developing the lip reading technology. This paper provides a comprehensive survey of the stateof-the-art deep learning based VSR research with a focus on data challenges, task-specific complications, and the corresponding solutions. Advancements in these directions will expedite the transformation of silent speech interface from theory to practice. We also discuss the main modules of a VSR pipeline and the influential datasets. Finally, we introduce some typical VSR application concerns and impediments to real-world scenarios as well as future research directions.

show abstract

A Survey of Lipreading Methods Based on Deep Learning

Cited by 5 publications

References 31 publications

FRIH: Fine-Grained Region-Aware Image Harmonization

FRIH: Fine-Grained Region-Aware Image Harmonization

Harmonized Portrait‐Background Image Composition

Advances and Challenges in Deep Lip Reading

Contact Info

Product

Resources

About