FusAtNet: Dual Attention based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDAR Classification

Abstract

With recent advances in sensing, multimodal data is becoming easily available for various applications, especially in remote sensing (RS), where many data types like multispectral imagery (MSI), hyperspectral imagery (HSI), LiDAR etc. are available. Effective fusion of these multisource datasets is becoming important, for these multimodality features have been shown to generate highly accurate land-cover maps. However, fusion in the context of RS is non-trivial considering the redundancy involved in the data and the large domain differences among multiple modalities. In addition, the feature extraction modules for different modalities hardly interact among themselves, which further limits their semantic relatedness. As a remedy, we propose a feature fusion and extraction framework, namely FusAtNet, for collective land-cover classification of HSIs and LiDAR data in this paper. The proposed framework effectively utilizses HSI modality to generate an attention map using "self-attention" mechanism that highlights its own spectral features. Similarly, a "cross-attention" approach is simultaneously used to harness the LiDAR derived attention map that accentuates the spatial features of HSI. These attentive spectral and spatial representations are then explored further along with the original data to obtain modality-specific feature embeddings. The modality oriented joint spectro-spatial information thus obtained, is subsequently utilized to carry out the land-cover classification task. Experimental evaluations on three HSI-LiDAR datasets show that the proposed method achieves the state-of-the-art classification performance, including on the largest HSI-LiDAR dataset available, University of Houston (Data Fusion Contest - 2013), opening new avenues in multimodal feature fusion for classification.

FunnyNet: Audiovisual Learning of Funny Moments in Videos

“…Discussion. CAF differs to existing methods [62,48] in the computation of the cross attention. Using stacked features F U to attend to each modality Q U brings three benefits: (a) it is order-agnostic: for any modality pair we compute crossattention once, instead of twice by interchanging queries and keys/values; this results in reduced computation; (b) each modality serves as a query to search for tokens in other modalities; this brings rich feature fusion; and (c) it generalizes to any number of modalities, resulting in scalability.…”

“…(a) Cross-attention is used in cross-domain knowledge transfer to learn acrosscue correlations by attending the features from one domain to another [62,48,63]. In CAF, it models the relationship among vision, audio, and face features.…”

Introduction to the Department of Cardiology in Nanjing First Hospital of Nanjing Medical University, China

“…Deep learning (DL), as an automatic feature learning technique, has demonstrated outstanding capabilities in feature extraction and has been widely applied to multimodal data fusion [3], [19], [20]. Among DL algorithms, convolutional neural networks (CNNs) are the go-to models for feature extraction due to their efficiency and ease of optimization [21].…”

“…In [25], [27], attention blocks are applied to a single modality, highlighting only unimodal features. Subsequently, some architectures [20], [28], [29] generate attention masks from one modality to enhance representations of the other. For example, FusAtNet [20] develops a "crossattention" module to exploit LiDAR-derived attention maps to highlight spatial features of HSIs.…”

“…Subsequently, some architectures [20], [28], [29] generate attention masks from one modality to enhance representations of the other. For example, FusAtNet [20] develops a "crossattention" module to exploit LiDAR-derived attention maps to highlight spatial features of HSIs. A self-and cross-guided attention model [28] refines information from HSI and LiDAR branches while providing cross-guidance from the LiDAR branch to the HSI branch.…”