FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

Abstract

Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.

Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis

“…These global representations are selected from the local features obtained by the attention mechanism. Although existing audiovisual fusion models are capable of obtaining effective joint representations, they are more complex and difficult to explain [5]. As described in [4], selecting key local components from the global representation is beneficial for reducing the complexity of the model.…”

“…Specifically, unimodal representations can only describe changes in emotion from a single perspective [4]. Therefore, compared with modeling unimodal information, previous research has focused on the use of specific deep neural networks (DNNs) to efficiently learn the joint representation of multiple modalities [5]. For instance, a large number of studies seek to tackle these challenges by building complex network structures [6] and fusing multimodal feature matrices [7], which can mine deep multimodal features and enhance interaction between audiovisual signals, respectively.…”

Application of a Fast RCNN Based on Upper and Lower Layers in Face Recognition

“…Xiaoning Zhang's paper proposes a new attention-guided network model that selectively integrates multilevel contextual information in an incremental manner. In addition to simulating the human attention mechanism, there is some research work that analyzes the importance of the information around the face object in judging the position of the face [29][30][31][32][33].…”

A multimodal molecular image fusion method based on relative total variation and co‐saliency detection

“…The saliency-based algorithm can make up for this deficiency well, and it can be combined with the above three categories of methods to improve the quality of the result images. 42 Ma et al defined pixel-level saliency to fuse base layers. 43 Zhang et al improved it by squaring the intensity difference to alleviate the problem of poor perception of lesions and edges.…”

“…The reason is that the characteristics of the image content itself and the sensitivity of the human eye to information, such as high brightness and contrast changes, are not better utilized. The saliency‐based algorithm can make up for this deficiency well, and it can be combined with the above three categories of methods to improve the quality of the result images 42 . Ma et al defined pixel‐level saliency to fuse base layers 43 .…”