Deep Exemplar-Based Video Colorization

Zhang, Bo; He, Mingming; Liao, Jing; Sander, Pedro V.; Liu, Yuan; Bermak, Amine; Chen, Dong

doi:10.1109/cvpr.2019.00824

Cited by 170 publications

(149 citation statements)

References 65 publications

Supporting

Mentioning

137

Contrasting

Order By: Relevance

“…The colors of output images often appear unnatural when given images not similar to input. In order to generalize to more reference images, He et al [23] and Zhang et al [52] applied deep image analogy technique and neural network to match the semantics of the target image and reference accurately. In addition, researchers used more types of references as guidance for colorization such as words [53], [54] and complete sentence [55].…”

Section: Related Workmentioning

confidence: 99%

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Zhao

Cheung

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Given a grayscale photograph, the colorization system estimates a visually plausible colorful image. Conventional methods often use semantics to colorize grayscale images. However, in these methods, only classification semantic information is embedded, resulting in semantic confusion and color bleeding in the final colorized image. To address these issues, we propose a fully automatic Saliency Map-guided Colorization with Generative Adversarial Network (SCGAN) framework. It jointly predicts the colorization and saliency map to minimize semantic confusion and color bleeding in the colorized image. Since the global features from pre-trained VGG-16-Gray network are embedded to the colorization encoder, the proposed SCGAN can be trained with much less data than state-of-the-art methods to achieve perceptually reasonable colorization. In addition, we propose a novel saliency map-based guidance method. Branches of the colorization decoder are used to predict the saliency map as a proxy target. Moreover, two hierarchical discriminators are utilized for the generated colorization and saliency map, respectively, in order to strengthen visual perception performance. The proposed system is evaluated on ImageNet validation set. Experimental results show that SCGAN can generate more reasonable colorized images than state-of-the-art techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Zhao

Cheung

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Thus, their results are not always good enough for solving our problem. He et al (He et al 2018; and Zhang et al (Zhang et al 2019) propose deep learning based algorithms for image and video colorization. But, they assume the pair of images are visually very different but semantically similar.…”

Section: Related Workmentioning

confidence: 99%

Cycle-CNN for Colorization towards Real Monochrome-Color Camera Systems

Xuan

Wang

et al. 2020

AAAI

View full text Add to dashboard Cite

Colorization in monochrome-color camera systems aims to colorize the gray image IG from the monochrome camera using the color image RC from the color camera as reference. Since monochrome cameras have better imaging quality than color cameras, the colorization can help obtain higher quality color images. Related learning based methods usually simulate the monochrome-color camera systems to generate the synthesized data for training, due to the lack of ground-truth color information of the gray image in the real data. However, the methods that are trained relying on the synthesized data may get poor results when colorizing real data, because the synthesized data may deviate from the real data. We present a new CNN model, named cycle CNN, which can directly use the real data from monochrome-color camera systems for training. In detail, we use the colorization CNN model to do the colorization twice. First, we colorize IG using RC as reference to obtain the first-time colorization result IC. Second, we colorize the de-colored map of RC, i.e. RG, using the first-time colorization result IC as reference to obtain the second-time colorization result R′C. In this way, for the second-time colorization result R′C, we use the original color map RC as ground-truth and introduce the cycle consistency loss to push R′C ≈ RC. Also, for the first-time colorization result IC, we propose a structure similarity loss to encourage the luminance maps between IG and IC to have similar structures. In addition, we introduce a spatial smoothness loss within the colorization CNN model to encourage spatial smoothness of the colorization result. Combining all these losses, we could train the colorization CNN model using the real data in the absence of the ground-truth color information of IG. Experimental results show that we can outperform related methods largely for colorizing real data.

show abstract

“…SG-Net works on feature correspondence calculation and utilization. We adopt the correlation matrix [11] to build 978-1-6654-3864-3/21/$31.00 ©2021 IEEE feature correspondence between the structural prior and the known region. Based on this idea, structural prior and input image are first processed by the domain alignment module, and correlation matrix is computed between aligned features.…”

Section: Introductionmentioning

confidence: 99%

Structural Prior Guided Image Inpainting for Complex Scene

Wei

Gao

2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

One of the main challenges faced by many existing image inpainting methods is that they fail to generate clear boundaries between different objects in a complex scene. To address this problem, we propose a novel two-stage image completion framework with semantic segmentation maps as the structure prior to restrain texture generation. Our framework consists of a PSP-based adversarial network called SP-Net, which can generate a semantic segmentation map for the corrupted image and an SG-Net using domain alignment and correlation matrix to get the feature correspondence between structural prior and known region. Combining the structural prior and feature correspondence, the image completion network fills the missing region with distinct boundaries and fine details. The proposed method was evaluated over Outdoor Scenes and Cityscapes datasets. Experimental results demonstrate that our approach outperforms other structural prior based approaches in terms of visual quality and quantitative measurements.

show abstract

Deep Exemplar-Based Video Colorization

Cited by 170 publications

References 65 publications

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Cycle-CNN for Colorization towards Real Monochrome-Color Camera Systems

Structural Prior Guided Image Inpainting for Complex Scene

Contact Info

Product

Resources

About