Bi-level Feature Alignment for Versatile Image Translation and Manipulation

Abstract

Generative adversarial networks (GANs) have achieved great success in image translation and manipulation. However, high-fidelity image generation with faithful style control remains a grand challenge in computer vision. This paper presents a versatile image translation and manipulation framework that achieves accurate semantic and style guidance in image generation by explicitly building a correspondence. To handle the quadratic complexity incurred by building the dense correspondences, we introduce a bi-level feature alignment strategy that adopts a top-k operation to rank block-wise features followed by dense attention between block features which reduces memory cost substantially. As the top-k operation involves index swapping which precludes the gradient propagation, we propose to approximate the non-differentiable top-k operation with a regularized earth mover's problem so that its gradient can be effectively back-propagated. In addition, we design a novel semantic position encoding mechanism that builds up coordinate for each individual semantic region to preserve texture structures while building correspondences. Further, we design a novel confidence feature injection module which mitigates mismatch problem by fusing features adaptively according to the reliability of built correspondences. Extensive experiments show that our method achieves superior performance qualitatively and quantitatively as compared with the state-of-the-art. The code is available at https://github.com/fnzhan/RABIT.

VMRF: View Matching Neural Radiance Fields

“…For example, Liu et al [19] exploit optimal transport in a unified framework to model semantic correspondences. For tackling unbalanced distributions with different masses and deviations, unbalanced optimal transport [4,15] has also been explored for different tasks, such as image translation [36,37,41]. To the best of our knowledge, the proposed VMRF the first that adapt unbalanced optimal transport for optimizing NeRF representations.…”

DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation

“…Wang et al [53] only involve tokens with top K attention scores for feature aggregation. Similar strategy is also adopted in [69,63,44]. Although effective, it is not flexible enough to fix the number of attentive tokens, which fails to model the complex and changeable matching patterns in practice.…”

“…To alleviate this issue, the works of [69,63] propose to fix the number of feature points that each query position focuses on, thereby achieving a linearcomplexity model. The reason behind such a design lies in that, each query position in target images is, in reality, independent to and dissimilar to most points in the reference.…”

Auto-regressive Image Synthesis with Integrated Quantization

“…Conditional image generation has achieved remarkable progress by learning the mapping among data of different domains. To achieve high-fidelity yet flexible image generation, various conditional inputs have been adopted including semantic segmentation [12,48,35,59,62], scene layouts [42,65,18], key points [26,29,61,57], edge maps [12,55,56], etc. Recently, several studies explored to generate images with cross-modal guidance [58,53].…”