Plasticity-Stability Preserving Multi-Task Learning for Remote Sensing Image Retrieval

Abstract

Deep learning-based multi-task learning (MTL) methods have recently attracted attention for content-based image retrieval (CBIR) applications in remote sensing (RS). For a given set of tasks (e.g., scene classification, semantic segmentation, and image reconstruction), existing MTL methods employ a joint optimization algorithm on the direct aggregation of task-specific loss functions. Such an approach may provide limited CBIR performance when: 1) tasks compete or even distract each other; 2) one of the tasks dominates the whole learning procedure; or 3) characterization of each task is underperformed compared to single-task learning. This is mainly due to the lack of: 1) plasticity condition (which is associated with sensitivity to new information) or 2) stability condition (which is associated with protection from radical disruptions by new information) of the whole learning procedure. To avoid this issue, as a first time, we propose a novel plasticity-stability preserving MTL (PLASTA-MTL) approach to ensure the plasticity and the stability conditions of the whole learning procedure independently of the number and type of tasks. This is achieved by defining two novel loss functions. The first loss function is the plasticity preserving loss (PPL) function that aims to enforce the global image representation space to be sensitive to new information learned with each task. This is achieved by minimizing the difference of gradient magnitudes for the global representation and task-specific embedding spaces. The second loss function is the stability preserving loss (SPL) function that aims to protect the global representation space radically disrupted by a new task. This is achieved by minimizing the angular distances between the task gradients over global representation space. To effectively employ the proposed loss functions, we also introduce a novel sequential optimization algorithm. Experimental results show the effectiveness of the proposed approach compared to the state-of-the-art MTL methods in the context of CBIR.

Remote Sensing Image Retrieval in the Past Decade: Achievements, Challenges, and Future Directions

“…Zhang [69]. Sumbul et al have proposed a novel plasticity-stability preserving multi-task learning approach to ensure the plasticity and the stability conditions of the whole learning procedure independently of the number and type of tasks [70].…”

Guided Deep Learning by Subaperture Decomposition: Ocean Patterns from SAR Imagery

“…However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering, the features being learned automatically from data. More closely to our task, there have been several works for content based image retrieval from remote sensing data [1], [5], [4], [39], [40]. In [4] authors propose a classical approach for EO image retrieval based on enriched metadata, semantic annotations and image content.…”

“…Ye et al [39] propose an unsupervised domain adaptation model based on convolutional neural networks (CNNs) to learn the domain-invariant feature between SAR images and optical aerial images for SAR image retrieving. In [5] authors propose a plasticitystability preserving multi-task learning approach to ensure the plasticity and the stability conditions of whole learning procedure independently from the number and type of tasks. This is achieved by defining two novel loss functions, the plasticity preserving loss and the stability preserving loss.…”

“…Usually, the key information is sparsely hidden in massive ocean-satellite data. Once with the growing capacity of collecting ocean data, many efforts have been put into developing and validating retrieval algorithms to generate standard time series global ocean parameters [1], [4], [5], [6], [7], [7], [8]. Currently, one important focus is to develop efficient and intelligent approaches to improve the information extraction capability with powerful deep learning algorithms.…”

Query by Example in Remote Sensing Image Archive Using Enhanced Deep Support Vector Data Description

“…For example, Cao et al [52] have provided an enhancement method for RS image retrieval by exploiting a triplet loss in learning discriminative deep features. Recently, Sumbul and Demir [53] have provided a DL-based multitask learning (MTL) for CBIR on DLRSC [54] and BigEarthNet [55] archives. They propose a novel enhancement of MTL by defining two novel loss functions to ensure the plasticity and stability conditions of the whole learning procedure independently of the number and type of tasks.…”