Visual attribute transfer through deep image analogy

Abstract

We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure. By visual attribute transfer, we mean transfer of visual information (such as color, tone, texture, and style) from one image to another. For example, one image could be that of a painting or a sketch while the other is a photo of a real scene, and both depict the same type of scene. Our technique finds semantically-meaningful dense correspondences between two input images. To accomplish this, it adapts the notion of "image analogy" [Hertzmann et al. 2001] with features extracted from a Deep Convolutional Neutral Network for matching; we call our technique deep image analogy. A coarse-to-fine strategy is used to compute the nearest-neighbor field for generating the results. We validate the effectiveness of our proposed method in a variety of cases, including style/texture transfer, color/style swap, sketch/painting to photo, and time lapse.

StyleBlit: Fast Example‐Based Stylization with Local Guidance

“…Although it produces impressive results for some style exemplars, it was shown to suffer from certain high‐frequency artifacts caused by the parametric nature of the synthesis algorithm [FJL∗16, FJS∗17]. To prevent texture distortion, researchers have proposed techniques to combine the advantages of patch‐based synthesis and the deep features learned by neural network [LW16, LYY∗17]. These approaches, however, have significant computational overhead and are not suitable for real‐time applications.…”

Deep Painterly Harmonization

“…Discussion . Our constrained mapping was inspired by the nearest‐neighbor field upsampling used in the Deep Analogy work [LYY*17, §4.4] that constrains the matches at each layer to come from a local region around the location at a previous layer. When the input and style images are similar, this technique performs well.…”

“…Our method automatically harmonizes the compositing of an element into a painting. Given the proposed painting and element on the left, we show the compositing results (cropped for best fit) of unadjusted cut‐and‐paste, Deep Image Analogy [LYY*17], and our method.…”

“…Recent approaches replace the Gram matrix with matching other statistics of neural responses [HB17,LFY*17]. Liao et al [LYY*17] further improve the quality of the results by introducing bidirectional dense correspondence field matching. All these methods have in common that they change the style of entire images at once.…”

Watercolour Rendering of Portraits

“…However, the training and test images are not truly watercolour style. A recent work attempts to combine both the Image Analogies and deep learning approaches [17], but is computationally expensive. Such example based approaches have some limitations.…”