Diversified Arbitrary Style Transfer via Deep Feature Perturbation

Wang, Zhizhong; Zhao, Lei; Chen, Haibo; Qiu, Lihong; Mo, Qihang; Lin, Sihuan; Xing, Wei; Lu, Dongming

doi:10.1109/cvpr42600.2020.00781

Cited by 95 publications

(72 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these methods have limited diversity. Another closely related work is [21], where a random orthogonal matrix has been introduced to modify the original whitening and coloring transformations in WCT [22]. This dependency leads to that the algorithm in [21] can be applied only to WCT-based frameworks.…”

Section: Diversitymentioning

confidence: 99%

“…Specifically, we concentrate on the style transfer methods which contain the idea of feature transformation, e.g., [36], [22], and [37]. Compared with [21], our style permutation algorithm does not amend the original whitening and coloring transformations. Instead, we modify the results generated by the whitening and coloring operations.…”

Section: Diversitymentioning

confidence: 99%

“…One can even insert orthogonal transformations into (5) and (6) at specified locations to obtain other solutions. And this is the key idea of the work in [21].…”

Section: A Background Of Linear Feature Transforma-tionsmentioning

confidence: 99%

“…Although their method can produce diverse outputs, the network has been trained on limited visual styles. Recently, Wang et al [21] proposed to insert an orthogonal random noise matrix to perturb the deep image feature maps to improve diversity. The major limitation is that their work highly relies on the system of [22], tending to lose the structure information of the content image.…”

Section: Introductionmentioning

confidence: 99%

“…It is worth noting that different feature maps could have the same Gram matrix, and the corresponding reconstructed images are what we need. Inspired by the work of Wang et al [21], which proposed to insert an orthogonal random noise matrix to perturb the deep image feature maps to improve diversity, we apply a specific orthogonal transformation on the deep transformed feature maps directly to tackle the limitation of diversity while retaining the original style information. The transformed feature maps that play the role of input data for a decoder have the same Gram matrix.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Style Permutation for Diversified Arbitrary Style Transfer

Zhang

Zhao

et al. 2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

Arbitrary neural style transfer aims to render a content image in a randomly given artistic style using the features extracted from a well-trained convolutional neural network. Existing style transfer algorithms have demonstrated astonishing results. However, the generated images suffer from loss of content details, non-uniform stroke patterns, and limited diversity. In this paper, we focus on improving the diversity of the stylized images. We propose a light-weighted yet efficient method named style permutation (SP) to tackle the limitation of the diversity without harming the original style information. The core of our style permutation algorithm is to multiply the deep image feature maps by a permutation matrix. Compared with state-of-the-art diversified style transfer methods, our style permutation algorithm offers more flexibility. Also, we present qualitative and quantitative analysis and theory explanations of the effectiveness of our proposed method. Experimental results show that our proposed method could generate diverse outputs for arbitrary styles when integrated into both WCT (whitening and coloring transform)-based methods and AdaIN (adaptive instance normalization)-based methods.

show abstract

Section: Diversitymentioning

confidence: 99%

Section: Diversitymentioning

confidence: 99%

“…One can even insert orthogonal transformations into (5) and (6) at specified locations to obtain other solutions. And this is the key idea of the work in [21].…”

Section: A Background Of Linear Feature Transforma-tionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Style Permutation for Diversified Arbitrary Style Transfer

Zhang

Zhao

et al. 2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

PKD‐Net: Distillation of prior knowledge for image completion by multi‐level semantic attention

Lu,

Lin,

Xing

et al. 2023

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SummaryPrior knowledge plays a crucial role in image completion. Although almost all of the existing image completion methods use prior knowledge to complete the image to be repaired from different perspectives, the learning and modeling of the prior knowledge is still a challenging problem. In order to address this issue, we propose a novel prior knowledge distillation framework (PKD‐Net) which could distill prior knowledge of structure and style from multiple semantic space and generates not only plausible content but also consistent style with surrounding image area. Our PKD‐Net replaces the skip connection in the vanilla U‐Net with a semantic shift attention module. The semantic shift attention module takes features from encoder layer and those from decoder layer as input pairs to output shifted features which take into account the long‐range dependency of encoder layer features and corresponding decoder layer features from the perspective of local structure and style. Semantic shift attention module models the global interdependencies in local spatial structures (patches centered at each position) and style (appearance texture) dimensions respectively, which could implement distillation of prior knowledge from two aspects: structure and style. Experiments on multiple datasets including faces (CelebA, CelebA‐HQ) and natural images (ImageNet, Places2, Paris Street View) demonstrate that our proposed approach generates higher quality completion results than existing ones.

show abstract

An illustration of model agnostic explainability methods applied to environmental data

et al. 2022

View full text Add to dashboard Cite

Historically, two primary criticisms statisticians have of machine learning and deep neural models is their lack of uncertainty quantification and the inability to do inference (i.e., to explain what inputs are important). Explainable AI has developed in the last few years as a sub‐discipline of computer science and machine learning to mitigate these concerns (as well as concerns of fairness and transparency in deep modeling). In this article, our focus is on explaining which inputs are important in models for predicting environmental data. In particular, we focus on three general methods for explainability that are model agnostic and thus applicable across a breadth of models without internal explainability: “feature shuffling”, “interpretable local surrogates”, and “occlusion analysis”. We describe particular implementations of each of these and illustrate their use with a variety of models, all applied to the problem of long‐lead forecasting monthly soil moisture in the North American corn belt given sea surface temperature anomalies in the Pacific Ocean.

show abstract

Diversified Arbitrary Style Transfer via Deep Feature Perturbation

Cited by 95 publications

References 20 publications

Style Permutation for Diversified Arbitrary Style Transfer

Style Permutation for Diversified Arbitrary Style Transfer

PKD‐Net: Distillation of prior knowledge for image completion by multi‐level semantic attention

An illustration of model agnostic explainability methods applied to environmental data

Contact Info

Product

Resources

About