Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation

Abstract

Fried chicken Cupcake Pineapple Strawberry Moon Cookie Orange Watermelon Soccer Basketball Class-conditioned Outline-to-Image Translation Interactive Sketch & FillFigure 1: (Top) Given a user created incomplete object outline (first row), our model estimates the complete shape and provides this as a recommendation to the user (shown in gray), along with the final synthesized object (second row). These estimates are updated as the user adds (green) or removes (red) strokes over time -previous edits are shown in black.(Bottom) This generation is class-conditioned, and our method is able to generate distinct multiple objects for the same outline (e.g. 'circle') by conditioning the generator on the object category. AbstractWe propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects. As the user starts to draw a sketch of a desired object type, the network interactively recommends plausible completions, and shows a corresponding synthesized image to the user. This enables a feedback loop, where the user can edit their sketch based on the network's recommendations, visualizing both the completed shape and final rendered image while they draw. In order to use a single trained model across a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network.

Image Search With Text Feedback by Visiolinguistic Attention Learning

“…Interactive image search aims to incorporate user feedback as an interactive signal to navigate the visual search. In general, the user interaction can be given in various formats, including relative attribute [45,28,75], attribute [79,18,2], attribute-like modification text [66], natural language [16,17], spatial layout [37], and sketch [76,74,14]. As text is the most pervasive interaction between human and computer in contemporary search engines, it naturally serves to convey concrete information that elaborates user's intricate specification for image search.…”

Semantic Image Analogy with a Conditional Single-Image GAN

“…GANs [12] have made a great success in image synthesis [5,18,19]. Conditional GANs synthesize images based on given conditions, which can be class labels [27], text [18], edges [11,16], or semantic segmentation maps [30,39]. Isola et al show the power of conditional GANs on generating images given dense condition signals including sketches and segmentation maps [16].…”

“…Generative models like Variational Autoencoders (VAEs) [21] and Generative Adversarial Networks (GANs) [12] have made great progress on modeling the distribution of natural images in a generative way. With additional signals such as class labels [27], text [44], edges [11], or segmentation maps [30,39] as input, conditional generative models can generate photorealistic samples in a controllable manner, which is useful in a number of multimedia applications such as interactive design [10,11,30] and artistic style transfer [8,43].…”

“…Comparisons with existing methods show the superiority of our method in terms of both quantitative and qualitative evaluations. Thanks to our flexible task setting, the proposed method can easily extend to various applications including object removal [14,41,42], face editing [22], and sketchto-image synthesis [11] for images in the wild.…”

Deep image synthesis from intuitive user input: A review and perspectives

“…An interesting next pursuit would be to see if computers can mimic creative processes such as those used by painters in making pictures, or assisting artists or architects in making artistic or architectural designs. In fact, in the past decade, we have witnessed advances in systems that synthesize an image from a text description [1][2][3][4] or from a learned style of content [5], paint a picture given a sketch [6][7][8][9], render a photorealistic scene from a wireframe [10,11], and create virtual reality content from images and videos [12], among others. A comprehensive review of such systems can explain the current state-of-the-art in such pursuits, reveal open challenges, and illuminate future directions.…”