Semi-supervised Learning for Face Sketch Synthesis in the Wild

Abstract

Face sketch synthesis has made great progress in the past few years. Recent methods based on deep neural networks are able to generate high quality sketches from face photos. However, due to the lack of training data (photo-sketch pairs), none of such deep learning based methods can be applied successfully to face photos in the wild. In this paper, we propose a semi-supervised deep learning architecture which extends face sketch synthesis to handle face photos in the wild by exploiting additional face photos in training. Instead of supervising the network with ground truth sketches, we first perform patch matching in feature space between the input photo and photos in a small reference set of photo-sketch pairs. We then compose a pseudo sketch feature representation using the corresponding sketch feature patches to supervise our network. With the proposed approach, we can train our networks using a small reference set of photo-sketch pairs together with a large face photo dataset without ground truth sketches. Experiments show that our method achieves state-of-the-art performance both on public benchmarks and face photos in the wild. Codes are available at https://github.com/chaofengc/Face-Sketch-Wild.

Multi-Scale Feature Channel Attention Generative Adversarial Network for Face Sketch Synthesis

“…Then, the attention mechanism is applied to learn the interdependencies of different features. Instead of training the generator network with the real sketches directly, the pseudo sketch feature [18], [41] loss from the pre-train VGG-19 network is utilized to supervise the synthesis. Total variation loss is also utilized to suppress noise in the generated images.…”

“…The features from the VGG-19 network have been proved to be effective for image matching [18], [41]. Following the FSWild method [18], the VGG-based feature representation [18], [41] is utilized to represent the features of an entire image. We divide the feature mappings from the VGG network to the patch level as the local features.…”

“…As we can get the MRF VGG-19 feature representation of pseudo sketch for any input image, the difference between MRF features of generated fake sketchŝ and the reference sketch [18], [41] is adopted as the main loss of the generator. Let Φ(ŝ) denote the feature mapping of fake sketchŝ extracted from the VGG-19 network and Ψ(Φ(ŝ)) denote the all the local overlapping neural patches extracted from Φ(ŝ).…”

“…In this paper, we aim to develop a fast face sketch synthesis method that exploits the characteristics of the sketches and produces more detailed sketch images with specific textures. Follow the FSWild [18], we combine deep learning networks with traditional patch-based methods and benefit from both of them. The GAN is adopted as the main framework to generate sketch images.…”

“…(3) The patch-wise feature loss from the pooling layers is proved to be robust in generating sketch-like images. Compared with the feature loss calculated from the ReLU layers [18], it can accelerate the training process by 3 times.…”