DoveNet: Deep Image Harmonization via Domain Verification

Cong, Wenyan; Zhang, Jianfu; Niu, Li; Liu, Liu; Ling, Zhixin; Li, Weiyuan; Zhang, Liqing

doi:10.1109/cvpr42600.2020.00842

Cited by 171 publications

(295 citation statements)

References 37 publications

Supporting

Mentioning

286

Contrasting

Order By: Relevance

“…Therefore, making the generated composite realistic could be a challenging task. Image harmonization [1,2,3], which aims to adjust the foreground to make it compatible with the background, is essential to address this problem. Traditional harmonization methods [4,5,6] improve the quality of synthesized composite mainly by transferring hand-crafted appearance statistics between foreground and background regions, but they could not handle the large appearance gap between foreground and background regions.…”

Section: Introductionmentioning

confidence: 99%

“…In [2], the spatial-separated attention blocks were proposed to learn the foreground and background features separately. Later in [3], they proposed an ad-versarial network with a domain verification discriminator to pull close the domains of foreground and background regions. Nonetheless, previous deep learning based methods neglected the crucial guidance role that background plays in the harmonization task.…”

Section: Introductionmentioning

confidence: 99%

“…According to DoveNet [3], we can treat different capture conditions as different domains. As illustrated in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…For simplicity, we spatially replicate the background domain code to the same size as input image and concatenate them along the channel dimension. The concatenated input, together with the foreground mask, is fed into an attention-enhanced U-net generator [3] to produce the harmonized result. At the same time, we propose two well-tailored triplet losses to ensure that the domain code extractor can indeed extract domain information instead of domain-irrelevant information (e.g., semantic layout).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Bargainnet: Background-Guided Domain Translation for Image Harmonization

Cong

Niu

Zhang

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

Self Cite

View full text Add to dashboard Cite

Given a composite image with inharmonious foreground and background, image harmonization aims to adjust the foreground to make it compatible with the background. Previous image harmonization methods mainly focus on learning the mapping from composite image to real image, while ignoring the crucial guidance role that background plays. In this work, we formulate image harmonization task as background-guided domain translation. Specifically, we use a domain code extractor to capture the background domain information to guide the foreground harmonization, which is regulated by well-tailored triplet losses. Extensive experiments on the benchmark dataset demonstrate the effectiveness of our proposed method. Code is available at https://github.com/bcmi/BargainNet.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…According to DoveNet [3], we can treat different capture conditions as different domains. As illustrated in Fig.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bargainnet: Background-Guided Domain Translation for Image Harmonization

Cong

Niu

Zhang

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

Self Cite

View full text Add to dashboard Cite

show abstract

“…One class of methods is harmonization by adjusting brightness and hue of foreground to match the scene. Existing methods include matching statistics [2][3][4], gradient-based blending [5,6], learning-based methods [7][8][9][10]. These methods do compositing in an end-to-end manner without explicit reconstruction of lighting, geometry and albedo.…”

Section: Related Workmentioning

confidence: 99%

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

Shen

Hou

et al. 2021

Sci. China Inf. Sci.

View full text Add to dashboard Cite

We present neural compositing, a deep-learning based method for augmented reality rendering, which uses convolutional neural networks to composite rendered layers of a virtual object with a real photograph to emulate shadow and reflection effects. The method starts from estimating the lighting and roughness information from the photograph using neural networks, renders the virtual object with a virtual floor into color, shadow and reflection layers by applying the estimated lighting, and finally refines the reflection and shadow layers using neural networks and blends them with the color layer and input image to yield the output image. We assume low-frequency lighting environments and adopt PRT (precomputed radiance transfer) for layer rendering, which makes the whole pipeline differentiable and enables fast end-to-end network training with synthetic scenes. Working on a single photograph, our method can produce realistic reflections in a real scene with spatially-varying material and cast shadows on background objects with unknown geometry and material at real-time frame rates.

show abstract

Analyzing the color availability of AI‐generated posters based on K‐means clustering: 74% orange, 38% cyan, 32% yellow, and 28% blue‐cyan

Rong,

Hansopaheluwakan‐Edward,

2023

Color Research & Application

View full text Add to dashboard Cite

In this exploratory study, we delved deeply into the intricate interplay of color choices within AI‐generated and human‐designed posters, analyzing a sample of 120 instances from each category. While it is suggested that human designers may integrate cultural, emotional, and situational contexts into their creations, AI models largely base their selections on vast datasets and pattern recognition. Although AI exhibited prowess in replicating established design parameters, the study underlined the importance of critically assessing its outputs. The quantitative analysis illuminated overarching similarities in primary color selections. However, the AI's diversity in color remains less concentrated than that of human, suggesting a gap in the AI's capacity to match human expertise in color proportioning and distribution. As AI continues to evolve, it is crucial to discern its capabilities and potential limitations in the design domain, ensuring it augments human creativity rather than supplanting it. Notably, the research refrains from seeking human validation, aiming instead for an objective, data‐driven reflection on the convergences and divergences between AI‐generated and human designs.

show abstract

DoveNet: Deep Image Harmonization via Domain Verification

Cited by 171 publications

References 37 publications

Bargainnet: Background-Guided Domain Translation for Image Harmonization

Bargainnet: Background-Guided Domain Translation for Image Harmonization

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

Analyzing the color availability of AI‐generated posters based on K‐means clustering: 74% orange, 38% cyan, 32% yellow, and 28% blue‐cyan

Contact Info

Product

Resources

About