Semantic colorization with internet images

Abstract

Colorization of a grayscale photograph often requires considerable effort from the user, either by placing numerous color scribbles over the image to initialize a color propagation algorithm, or by looking for a suitable reference image from which color information can be transferred. Even with this user supplied data, colorized images may appear unnatural as a result of limited user skill or inaccurate transfer of colors. To address these problems, we propose a colorization system that leverages the rich image content on the internet. As input, the user needs only to provide a semantic text label and segmentation cues for major foreground objects in the scene. With this information, images are downloaded from photo sharing websites and filtered to obtain suitable reference images that are reliable for color transfer to the given grayscale photo. Different image colorizations are generated from the various reference images, and a graphical user interface is provided to easily select the desired result. Our experiments and user study demonstrate the greater effectiveness of this system in comparison to previous techniques.

BiggerPicture

“…It has proven successful for e.g. object and scene recognition [Torralba et al 2008], image completion [Hays and Efros 2007], exploring visual space [Sivic et al 2008;Kaneva et al 2010], scene parsing [Liu et al 2009], image montage [Chen et al 2009;Chen et al 2013] and image enhancement [Dale et al 2009;Chia et al 2011;HaCohen et al 2013;Shih et al 2013]. A comprehensive summarization of data-driven approaches can be found in [Hu et al 2013a].…”

“…A second class of approach carefully filters out irrelevant images step by step [Chen et al 2009;Chia et al 2011]. At each step, undesirable images are discarded, leaving a smaller and smaller set of candidate images until a tractable set remains.…”

“…Sketch2Photo [Chen et al 2009] uses saliency, contour consistency and content consistency criteria to find foreground objects that are then merged with background images to make a photo montage. Chia et al [2011] use a similar approach to retrieve images for colorization. Our graphmatching process is also a cascading filtering procedure that considers high-level structure as well as low-level image features such as color, texture and shape of image regions near the extrapolating boundaries, to ensure natural extrapolation.…”

“…These chosen images provide components for extrapolation. Similar user interaction is widely applied in data-driven approaches [Hays and Efros 2007;Sivic et al 2008;Chen et al 2009;Kaneva et al 2010;Chia et al 2011].…”

A framework for interactive image color editing

“…Shapira et al [27] proposed a solution which is based on navigation through the appearance of the image in order to obtain desired results. Also automatic methods to colorize grayscale images based on examples from internet images [16], and semantic annotations [8] have been introduced. In general, methods which transfer colors globally are not suitable for precise (re-)coloring of small objects or humans.…”