Two-stage sketch colorization

Abstract

Sketch or line art colorization is a research field with significant market demand. Different from photo colorization which strongly relies on texture information, sketch colorization is more challenging as sketches may not have texture. Even worse, color, texture, and gradient have to be generated from the abstract sketch lines. In this paper, we propose a semi-automatic learning-based framework to colorize sketches with proper color, texture as well as gradient. Our framework consists of two stages. In the first drafting stage, our model guesses color regions and splashes a rich variety of colors over the sketch to obtain a color draft. In the second refinement stage, it detects the unnatural colors and artifacts, and try to fix and refine the result. Comparing to existing approaches, this two-stage design effectively divides the complex colorization task into two simpler and goal-clearer subtasks. This eases the learning and raises the quality of colorization. Our model resolves the artifacts such as water-color blurring, color distortion, and dull textures. We build an interactive software based on our model for evaluation. Users can iteratively edit and refine the colorization. We evaluate our learning model and the interactive system through an extensive user study. Statistics shows that our method outperforms the state-of-art techniques and industrial applications in several aspects including, the visual quality, the ability of user control, user experience, and other metrics.

Active Colorization for Cartoon Line Drawings

“…[2] propagated a user's scribbles throughout relevant regions in the image by using the level-set method. Recently, deep learning based methods have been proposed for colorization in either automatic or interactive manners [8], [9], [10], [11]. However, these methods may generate color strings along the border and tend to color the image differently from what the user specified.…”

“…However, when applied to discontinuous sketch images, their method struggles to find accurate correspondences. Researchers have applied deep learning models [5], [6], [7] to colorize cartoon images using reference images. Furusawa et al [5] first described reference images using color histograms, before using this expression to determine how cartoon characters should be colored in subsequent frames.…”

“…This method produces results with less detail than those depicted in the reference image while also reducing the amount of shading present in the image. Zhang et al [6] used deep VGG features as a descriptor for the reference images, but their method produced unclear object boundaries and mixed colors. Hensman et al [7] used a conditional GAN to colorize grayscale images.…”

“…(g) are colorized by method [6] with reference images (a) and (f), respectively; (c) and (h) show the results of method [7], which are obtained by postprocessing; (d) and (i) are the results of method [12]. It can be seen that method [12] cannot handle cases when the character has large deformations or when the numbers of nodes between images differ greatly.…”

“…The number of interactions will increase linearly with the number of pictures that need to be colorized. Recent cartoon colorization works have focused on using deep learning methods to colorize sketch images automatically [5], [6], [7]. However, artifacts appear along the edges in these research works.…”