Disir: Deep Image Segmentation With Interactive Refinement

Abstract

Abstract. This paper presents an interactive approach for multi-class segmentation of aerial images. Precisely, it is based on a deep neural network which exploits both RGB images and annotations. Starting from an initial output based on the image only, our network then interactively refines this segmentation map using a concatenation of the image and user annotations. Importantly, user annotations modify the inputs of the network – not its weights – enabling a fast and smooth process. Through experiments on two public aerial datasets, we show that user annotations are extremely rewarding: each click corrects roughly 5000 pixels. We analyze the impact of different aspects of our framework such as the representation of the annotations, the volume of training data or the network architecture. Code is available at this address.

Interactive Learning for Semantic Segmentation in Earth Observation

“…Visual abstract of DISCA method. Up: initial prediction, middle: retraining using annotations as a sparse ground truth, bottom: final prediction using the retrained model in a DISIR [12] mode.…”

“…We rely on DISIR § [12] since it is, to the best of our knowledge, the only existing open-source work addressing multi-class interactive segmentation using deep learning in remote sensing. Hence, our network takes as input a concatenation of the RGB image and of the user annotations.…”

“…In this section, we study through experiments the scope of our approach and compare it to [12]. We experiment on three semantic segmentation remote sensing datasets: the INRIA Aerial Image Labelling dataset [14] composed of two classes (buildings and not buildings) and covering more than 800 km 2 in different cities at a 30cm resolution, the Aerial Imagery for Roof Segmentation (AIRS) dataset [6] composed of the same two classes and covering 457 km 2 in New-Zealand at a 7.5cm resolution and the ISPRS Potsdam dataset [18] composed of 6 classes (impervious surface, buildings, low vegetation, tree, car and clutter ) covering 3 km 2 on Potsdam at a 5cm resolution.…”

“…We do not assume such extreme point map availability as it is costly to acquire in a remote sensing image with many potential objects. In our previous work [12], we extended [21] to interactively refine remote sensing segmentation maps. We now draw inspiration from [10] to the same purpose.…”

“…In practice, the user can work hand-in-hand with machine learning models which have been previously trained on large amounts of data. Indeed, as shown by DIOS [21] or DISIR [12], a user can guide a neural network to perform segmentation tasks with clicked annotations given as inputs to the algorithm. Since these approaches do not modify the weights of the networks, their guidance is spatially localized around the annotations.…”