Spatiotemporal Colorization of Video Using 3D Steerable Pyramids

Paul, Somdyuti; Bhattacharya, Sumanta; Gupta, Sumana

doi:10.1109/tcsvt.2016.2539539

Cited by 22 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first is to post-process the framewise colorization with a general temporal filter [21,22], but these works tend to wash out the colors. Another class of methods propagate the color scribbles to other frames by explicitly calculating the optical flow [1,2,23,24,25]. However, scribbles drawn from one specific image may not be suitable for other frames.…”

Section: Related Workmentioning

confidence: 99%

“…A naïve approach is to run a temporal filter on the per-frame colorization results during post-processing [21,22], which can alleviate the flickering but cause color fading and blurring. Another set of approaches propagate the color scribbles across frames using optical flow [1,2,23,24,25]. However, scribbles propagation may be not perfect due to flow error, which will induce some visual artifacts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Exemplar-Based Video Colorization

Zhang

Liao

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

170

137

View full text Add to dashboard Cite

This paper presents the first end-to-end network for exemplar-based video colorization. The main challenge is to achieve temporal consistency while remaining faithful to the reference style. To address this issue, we introduce a recurrent framework that unifies the semantic correspondence and color propagation steps. Both steps allow a provided reference image to guide the colorization of every frame, thus reducing accumulated propagation errors. Video frames are colorized in sequence based on the colorization history, and its coherency is further enforced by the temporal consistency loss. All of these components, learned end-to-end, help produce realistic videos with good temporal stability. Experiments show our result is superior to the state-of-the-art methods both quantitatively and qualitatively.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep Exemplar-Based Video Colorization

Zhang

Liao

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

170

137

View full text Add to dashboard Cite

show abstract

“…To address the ambiguity brought by sparse scribbles. Xu et al [12] proposed a novel approximation scheme requiring much less time and memory and Paul et al [51] proposed a 3D steerable pyramids approach for occlusion handling. Since the aforementioned methods require accurate scribbles for colorization, Zhang et al introduced an additional deep prior from a CNN to ensure plausible colorization when no given scribbles.…”

Section: Related Workmentioning

confidence: 99%

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Zhao

Cheung

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Given a grayscale photograph, the colorization system estimates a visually plausible colorful image. Conventional methods often use semantics to colorize grayscale images. However, in these methods, only classification semantic information is embedded, resulting in semantic confusion and color bleeding in the final colorized image. To address these issues, we propose a fully automatic Saliency Map-guided Colorization with Generative Adversarial Network (SCGAN) framework. It jointly predicts the colorization and saliency map to minimize semantic confusion and color bleeding in the colorized image. Since the global features from pre-trained VGG-16-Gray network are embedded to the colorization encoder, the proposed SCGAN can be trained with much less data than state-of-the-art methods to achieve perceptually reasonable colorization. In addition, we propose a novel saliency map-based guidance method. Branches of the colorization decoder are used to predict the saliency map as a proxy target. Moreover, two hierarchical discriminators are utilized for the generated colorization and saliency map, respectively, in order to strengthen visual perception performance. The proposed system is evaluated on ImageNet validation set. Experimental results show that SCGAN can generate more reasonable colorized images than state-of-the-art techniques.

show abstract

“…C OLOR transfer aims at transferring an expected color into a target image. Thus, the target image will present different color appearances similar to different expected colors [1], [2]. Color transfer is useful for computer vision, graphics and image processing/editing applications [3], [4], such as animation, aesthetic enhancement, image stitching, and color correction [5].…”

Section: Introductionmentioning

confidence: 99%

Fast Hue-Division-Based Selective Color Transfer

Panetta

Bao

Agaian

2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

This paper introduces a new concept, selective color transfer (SCT), to allow for color experimentation and visualization on a selected object within an image, without affecting any other image content or color. To implement this new concept, this paper proposes a new fast SCT-I algorithm for image input data and a new SCT-V algorithm for video input data. For the SCT-I algorithm, its fundamental novelties include: 1) utilizing a new hue-division-based color region segmentation (HCR) based on defining a new nonlinear and circular property of the hue spectrum; 2) developing two new color artifact suppression strategies, including color region integration and circular statistical calculation; and 3) applying a new truncated color transfer equation to solve the problem of mismatches in the data range. The SCT-V algorithm utilizes: 1) object tracking to locate the target of interest (TOI) within each video frame; 2) the SCT-I algorithm to update the original color within TOIs; and 3) an expected color interpolation process to support different special color animation effects. The experimental results demonstrate: 1) there are no residual color artifacts introduced;2) the approaches have tremendous flexibility for modifying an object's color; and 3) these algorithms have a low computational cost. Furthermore, these algorithms require minimal manual intervention for the easy use of ordinary users. The performance analysis demonstrates the potential of these two methods for various applications, including creating special effects in videos, making histology images much clearer for medical inspection, and enabling low-cost experimentation of manufacturing the design of a product.

show abstract

Spatiotemporal Colorization of Video Using 3D Steerable Pyramids

Cited by 22 publications

References 27 publications

Deep Exemplar-Based Video Colorization

Deep Exemplar-Based Video Colorization

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Fast Hue-Division-Based Selective Color Transfer

Contact Info

Product

Resources

About