Neural Style Transfer: A Review

Abstract

The seminal work of Gatys et al. demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style. This process of using CNNs to render a content image in different styles is referred to as Neural Style Transfer (NST). Since then, NST has become a trending topic both in academic literature and industrial applications. It is receiving increasing attention and a variety of approaches are proposed to either improve or extend the original NST algorithm. In this paper, we aim to provide a comprehensive overview of the current progress towards NST. We first propose a taxonomy of current algorithms in the field of NST. Then, we present several evaluation methods and compare different NST algorithms both qualitatively and quantitatively. The review concludes with a discussion of various applications of NST and open problems for future research. A list of papers discussed in this review, corresponding codes, pre-trained models and more comparison results are publicly available at: https://github.com/ycjing/Neural-Style-Transfer-Papers. IndexTerms-Neural style transfer (NST), convolutional neural network ! Neural Style Transfer Example-Based Techniques Colour Image Analogy Texture Model-Optimisation-Based Offline Neural Methods Multiple-Style-Per-Model Neural Methods Dumoulin'17 [53] Chen'17 [54] Li'17 [55] Zhang'17 [56] Luan'17 [84] Mechrez'17 [85]Photorealistic Liao'17 [88] Attribute Champandard'16 [65] Doodle Ruder'16 [74] Video Selim'16 [73] Portrait Castillo'17 [71] Instance Gatys'17 [60] Improvement Image Gatys'16 [10] Li'17 [42] Risser'17 [44] Li'17 [45] Li'16 [46] Image Champandard'16 [65] Chen'16 [68] Mechrez'18 [69]

GANILLA: Generative adversarial networks for image to illustration translation

“…The second approach utilizes GANs to synthesize stylized images. There are very comprehensive survey papers for both CNN [30] and GAN [31] based methods. Here we summarize the GAN based unpaired image-to-image translation methods which are the most relevant group of work to our model.…”

Recent Advances of Image Steganography With Generative Adversarial Networks

“…The GAN-based approach so far produces images that are sharper than any other generation method. Image translation is related to style transfer [71], which constructs a generated image with specific content and style by using a content image and a style image. Image-to-image translation by GANs has also been successfully applied in some image or video generation applications [72].…”

A Closed-Form Solution to Universal Style Transfer

“…A variety of methods on neural style transfer have been proposed since the seminal work of Gatys [8]. These methods can be roughly categorized into image optimization and model optimization [13]. Methods based on image optimization directly obtain the stylized output by minimizing the content loss and style loss.…”

“…Apart from image and model optimization, many other works study the problems of semantic style transfer [23,21,1], video style transfer [11,2,26,27], portrait style transfer [28], and stereoscopic style transfer [4]. [13] provides a thorough review of the works on style transfer.…”