Searching a Hierarchically Aggregated Fusion Architecture for Fast Multi-Modality Image Fusion

Abstract

Multi-modality image fusion refers to generating a complementary image that integrates typical characteristics from source images. In recent years, we have witnessed the remarkable progress of deep learning models for multi-modality fusion. Existing CNN-based approaches strain every nerve to design various architectures for realizing these tasks in an end-to-end manner. However, these handcrafted designs are unable to cope with the high demanding fusion tasks, resulting in blurred targets and lost textural details. To alleviate these issues, in this paper, we propose a novel approach, aiming at searching effective architectures according to various modality principles and fusion mechanisms. Specifically, we construct a hierarchically aggregated fusion architecture to extract and refine fused features from feature-level and object-level fusion perspectives, which is responsible for obtaining complementary target/detail representations. Then by investigating diverse effective practices, we composite a more flexible fusion-specific search space. Motivated by the collaborative principle, we employ a new search strategy with different principled losses and hardware constraints for sufficient discovery of components. As a result, we can obtain a task-specific architecture with fast inference time. Extensive quantitative and qualitative results demonstrate the superiority and versatility of our method against state-of-the-art methods. CCS CONCEPTS• Computing methodologies → Computer vision.

Guided Image Deblurring by Deep Multi-Modal Image Fusion

“…However, handcrafted architectures of fusion are mostly based on existing mechanisms, limited in the heavy labor and design experiences. On the other hand, the mainstream differentiable search strategies [20], [21] have been introduced with large-scale datasets, which cannot estimate the accurate gradients due to the one-step approximation considering the efficiency. Thus, these methods are easy to generate unstable architectures, especially for the insufficient data of image fusion.…”

“…Search Configurations. The search space of image fusion is introduced from [21], which provide various image fusion-oriented cells and operators. In [21], it provides the details of cells (i.e., successive cell C SC , decomposition cell C DC and multi-scale fusion cell C MS ).…”

“…The search space of image fusion is introduced from [21], which provide various image fusion-oriented cells and operators. In [21], it provides the details of cells (i.e., successive cell C SC , decomposition cell C DC and multi-scale fusion cell C MS ). The search space of operators including Channel Attention (CA), Spatial Attention (SA), Dilated Convolution (DC), Residual Block (RB), Dense Block (DB) and Separable Convolution (SC) with different kernel size (3 × 3 and 5 × 5) are provided at [21], [22].…”

“…In [21], it provides the details of cells (i.e., successive cell C SC , decomposition cell C DC and multi-scale fusion cell C MS ). The search space of operators including Channel Attention (CA), Spatial Attention (SA), Dilated Convolution (DC), Residual Block (RB), Dense Block (DB) and Separable Convolution (SC) with different kernel size (3 × 3 and 5 × 5) are provided at [21], [22]. We define the regularization Reg as the weighted summation of GPU latency, aiming to obtain the lightweight efficient architecture.…”

“…As for end-toend learning, dense connection [1], [12], [14], and residual modules [13], [19] are proposed to aggregate the modal feature jointly. Meanwhile, there are few works that leverage the differentiable architecture search [20] to discover the suitable architectures for image fusion [21], [22]. Although these methods achieve remarkable performance, the mainstream search strategies are always leveraged for the largescale dataset, sacrificing the accurate gradient estimation.…”