Language-based colorization of scene sketches

Abstract

It's a moonlit night. There is a moon in the sky. A house is in the middle. A car is in front of the house. Three trees are on the left of the house. Two trees are on the right back of the car. There is a road.It is a moonlit night. There is a yellow moon in the sky. An orange house with light gray roof is in the middle. A yellow car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a yellow road. The sky is blue and all things are on green grass.It is a moonlit night. There is a purple moon in the sky. A yellow house with yellow roof is in the middle. A blue car with light gray window is in front of the house. Three yellow trees are on the left of the house. Two yellow trees are on the right back of the car. There is a light brown road. The sky is gray and all things are on yellow grass.It is a moonlit night. There is a yellow moon in the sky. A blue house with blue roof is in the middle. A red car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a dark gray road. The sky is blue and all things are on green grass. Fig. 1. We present LUCSS, a language-based interactive colorization system for scene sketches. It takes advantage of both instance level segmentation and language models in a unified generative adversarial network, allowing users to accomplish different colorization goals in a form of language instructions. Left: input scene sketch and its content description automatically generated by LUCSS. Three right columns show the colorization results generated by LUCSS following three different instructions at the bottom. Texts underlined are user-specified, with the target colors highlighted in bold.Abstract. We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Finally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its components to alternative choices.

Generative Adversarial Networks–Enabled Human–Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends

“…All the design tasks that we have covered till now have been aimed at expert or semi-expert users. Zou et al (2019) presented an example of human–AI collaboration primarily designed for children. The system is trained on a set of scene sketches and cartoon-style color images with text descriptions.…”

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

“…In order to generalize to more reference images, He et al [23] and Zhang et al [52] applied deep image analogy technique and neural network to match the semantics of the target image and reference accurately. In addition, researchers used more types of references as guidance for colorization such as words [53], [54] and complete sentence [55]. However, the combination of examples and input grayscale image is difficult in terms of transferring examples to useful color information.…”

Text as Neural Operator:Image Manipulation by Text Instruction

“…To enable user-guided manipulation, a variety of frameworks [3,9,21,23,33,37,43,47,52,57,62,63,70,75,77,78,81] have been proposed to study different control signals. For instance, Zhang et al [78] and Zou et al [86] used sparse dots and text respectively to guide the image colorization process. There are additional works on image manipulation by bounding boxes subsequently refined as semantic masks [20] or code [49].…”