Semantic Context-Aware Image Style Transfer

Liao, Yi-Sheng; Huang, Chun-Rong

doi:10.1109/tip.2022.3149237

Cited by 25 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, it iteratively optimizes the content image to match the style of the referenced image. Since then, numerous methods have been proposed to enhance various aspects of performance, such as controllability [18][19][20]22], text-driven [2,3,11,12] and quality [24][25][26][27][28].…”

Section: Style Transfermentioning

confidence: 99%

ConIS: Controllable Text-Driven Image Stylization with Semantic Intensity

Yang,

Li,

Zhang

2024

Preprint

View full text Add to dashboard Cite

Text-driven image stylization aims to synthesize content images with learned textual styles. Recent studies have shown the potential of the diffusion model for producing rich stylizations. However, existing approaches inefficiently control the degree of stylization, which hinders the balance between style and content in generated images. In this paper, we propose a Controllable Text-Driven Image Stylization (ConIS) Framework based on the diffusion model. The proposed framework introduces two modules into the pre-trained text-to-image model. The first is an Unconditional Null-text Inversion (UNTI) module, which optimizes null-text embedding to reduce the bias between inversion and sampling in the diffusion model. Given a content image, this module is able to reconstruct it without semantic guidance. The second is a Null-text Dilution (NTD) module. We design a parameterization mechanism for the semantic intensity of textual conditions, which indirectly controls the degree of stylization through the style degree factor. Finally, we replace the attention maps used in the sampling process with those from the UNTI module to constrain the structure of content images. Experiments have shown that the proposed method enables fine-grained control over the degree of stylization without retraining or fine-tuning the network. Both qualitative and quantitative results indicate that the ConIS framework outperforms state-of-the-art methods in balancing artistic detail and content structure.

show abstract

Section: Style Transfermentioning

confidence: 99%

ConIS: Controllable Text-Driven Image Stylization with Semantic Intensity

Yang,

Li,

Zhang

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Yeh et al (2020) designed two statistical indicators of style validity and content consistency to evaluate the style transfer algorithm, which promotes the design of standardized evaluation methods. Liao and Huang (2022) proposed a new semantic context-aware IST approach by conducting semantic context matching adopted by hierarchical local to global network structure. Babaeizadeh and Ghiasi (2018) presented a method that could adjust and control the generated style images on the trained model in real time.…”

Section: Related Workmentioning

confidence: 99%

“…The scene structure and object boundary of the content image needs to be retained for a high-quality IST, and the appearance should be aligned with the style image. In recent years, IST technologies (Gatys et al, 2016;Ghiasi et al, 2017;Junginger et al, 2018;Karras et al, 2019;Li et al, 2018;Li & Wand, 2016;Liao & Huang, 2022;Qiao et al, 2021;Wang et al, 2020;Yao et al, 2019;Yeh et al, 2020;Zhang & Dana, 2017) based on deep learning (DL) have demonstrated that the relevance among features obtained by CNNs is very available for obtaining visual contents and styles, and is utilized to obtain images with similar contents and styles.…”

Section: Introductionmentioning

confidence: 99%

Circular LBP Prior-Based Enhanced GAN for Image Style Transfer

Wen-guang

2022

International Journal on Semantic Web and Information Systems

View full text Add to dashboard Cite

Image style transfer (IST) has drawn broad attention recently. At present, convolutional neural network (CNN)-based methods and generative adversarial network (GAN)-based methods have been broadly utilized in IST. However, the texture of images obtained by most methods presents a lower definition, which leads to insufficient details of IST. To this end, the authors present a new IST method based on an enhanced GAN with a prior circular local binary pattern (LBP). They utilize circular LBP in a GAN generator as a texture prior to improve the detailed textures of the generated style images. Meanwhile, they integrate a dense connection residual block and an attention mechanism into the generator to further improve high-frequency feature extraction. In addition, the total variation (TV) regularizer is integrated into the loss function to smooth the training results and restrain the noise. The qualitative and quantitative experimental results demonstrate that the metric quality of the generated images can achieve better effects by the proposed strategy compared with other popular approaches.

show abstract

“…However, many of these methods usually require an iterative process or slow post-processing that is computationally expensive. Prior works also explore style transfer using visual cues from semantic segmentation [10], [21], [22], [23], improving the photorealism based on the matting Laplacian [18], [24], screened Poisson equation [25], or photorealistic smoothing [15] as a post-processing step. Our method belongs to the family of techniques in which segmentation information is explicitly used to address the style transfer problem.…”

Section: Style Transfermentioning

confidence: 99%

Time-of-Day Neural Style Transfer for Architectural Photographs

Chen

Ka-Chun

et al. 2022

2022 IEEE International Conference on Computational Photography (ICCP)

View full text Add to dashboard Cite

Architectural photography is a genre of photography that focuses on capturing a building or structure in the foreground with dramatic lighting in the background. Inspired by recent successes in image-to-image translation methods, we aim to perform style transfer for architectural photographs. However, the special composition in architectural photography poses great challenges for style transfer in this type of photographs. Existing neural style transfer methods treat the architectural images as a single entity, which would generate mismatched chrominance and destroy geometric features of the original architecture, yielding unrealistic lighting, wrong color rendition, and visual artifacts such as ghosting, appearance distortion, or color mismatching. In this paper, we specialize a neural style transfer method for architectural photography. Our method addresses the composition of the foreground and background in an architectural photograph in a two-branch neural network that separately considers the style transfer of the foreground and the background, respectively. Our method comprises a segmentation module, a learning-based image-to-image translation module, and an image blending optimization module. We trained our image-to-image translation neural network with a new dataset of unconstrained outdoor architectural photographs captured at different magic times of a day, utilizing additional semantic information for better chrominance matching and geometry preservation. Our experiments show that our method can produce photorealistic lighting and color rendition on both the foreground and background, and outperforms general image-to-image translation and arbitrary style transfer baselines quantitatively and qualitatively. Our code and data are available at https://github.com/hkust-vgd/architectural style transfer.

show abstract

Semantic Context-Aware Image Style Transfer

Cited by 25 publications

References 29 publications

ConIS: Controllable Text-Driven Image Stylization with Semantic Intensity

ConIS: Controllable Text-Driven Image Stylization with Semantic Intensity

Circular LBP Prior-Based Enhanced GAN for Image Style Transfer

Time-of-Day Neural Style Transfer for Architectural Photographs

Contact Info

Product

Resources

About