Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition

Abstract

Aerial scene recognition is a fundamental task in remote sensing and has recently received increased interest. While the visual information from overhead images with powerful models and efficient algorithms yields considerable performance on scene recognition, it still suffers from the variation of ground objects, lighting conditions etc. Inspired by the multi-channel perception theory in cognition science, in this paper, for improving the performance on the aerial scene recognition, we explore a novel audiovisual aerial scene recognition task using both images and sounds as input. Based on an observation that some specific sound events are more likely to be heard at a given geographic location, we propose to exploit the knowledge from the sound events to improve the performance on the aerial scene recognition. For this purpose, we have constructed a new dataset named AuDio Visual Aerial sceNe reCognition datasEt (ADVANCE). With the help of this dataset, we evaluate three proposed approaches for transferring the sound event knowledge to the aerial scene recognition task in a multimodal learning framework, and show the benefit of exploiting the audio information for the aerial scene recognition. The source code is publicly available for reproducibility purposes.

MultiScene: A Large-Scale Dataset and Benchmark for Multiscene Recognition in Single Aerial Images

“…Recently, the emergence of deep CNNs brings immense advancements to the community, and many achievements [19]- [34] have been obtained in the field of aerial single-scene classification. These deep networks have hierarchical architectures, where convolutional and max-pooling layers are periodically interleaved for learning high-level features of intricate scenes.…”

“…Besides, exploiting supplementary data, such as geotagged audios and multitemporal images, has been a new research direction. Hu et al [19] proposed to predict scene categories by transferring sound event knowledge learned from sound-image pairs. Ru et al [25] proposed a two-branch network to learn deep features of bitemporal images and fused them through a CorrFusion module for aerial scene classification.…”

“…As one of the crucial steps toward these applications, aerial scene recognition has been extensively studied in the remote sensing community. During the last few years, the emergence of deep convolutional neural networks (CNNs) pushed ahead research in this field, and enormous achievements [19]- [26] have been obtained. Albeit successful, most existing scene classification studies only focus on a specific scenario, where an aerial image is assumed to include a single scene [27]- [34].…”

Toward a Collective Agenda on AI for Earth Science Data Analysis

“…Beyond videos, other interesting examples include VQA (see Direction 3, page 8), captioning [27] and audiovisual reasoning, i.e., linking remote sensing images to in-situ audio signals [28]. In the long run, we hope that reasoning Earth observation systems would be capable of deduce clues and make structural inference, in order to explain processes (see direction 5, page 13) and understand causal structures in Earth Systems (see direction 6, page 16).…”

Foley Music: Learning to Generate Music from Videos

“…Follow up works [2,33] further investigated to jointly learn the visual and audio representation using a visual-audio correspondence task. Instead of learning feature representations, recent works have also explored to localize sound source in images or videos [29,26,3,48,64], biometric matching [39], visual-guided sound source separation [64,15,19,60], auditory vehicle tracking [18], multi-modal action recognition [36,35,21], audio inpainting [66], emotion recognition [1], audio-visual event localization [56], multi-modal physical scene understanding [16], audio-visual co-segmentation [47], aerial scene recognition [27] and audio-visual embodied navigation [17].…”