Extending pretrained segmentation networks with additional anatomical structures

Abstract

For comprehensive surgical planning with sophisticated patient-specific models, all relevant anatomical structures need to be segmented. This could be achieved using deep neural networks given sufficiently many annotated samples, however datasets of multiple annotated structures are often unavailable in practice and costly to procure. Therefore, being able to build segmentation models with datasets from different studies and centers in an incremental fashion is highly desirable. We propose a class-incremental framework for extending a deep segmentation network to new anatomical structures using a minimal incremental annotation set. Through distilling knowledge from the current network state, we overcome the need for a full retraining. We evaluate our methods on 100 MR volumes from SKI10 challenge with varying incremental annotation ratios. For 50% incremental annotations, our proposed method suffers less than 1% Dice score loss in retaining old-class performance, as opposed to 25% loss of conventional finetuning. Our framework inherently exploits transferable knowledge from previously trained structures to incremental tasks, demonstrated by results superior even to non-incremental training: In a single volume one-shot incremental learning setting, our method outperforms vanilla network performance by >11% in Dice. With the presented method, new anatomical structures can be learned while retaining performance for older structures, without a major increase in complexity and memory footprint; hence suitable for lifelong class-incremental learning. By leveraging information from older examples, a fraction of annotations can be sufficient for incrementally building comprehensive segmentation models. With our meta-method, a deep segmentation network is extended with only a minor addition per structure, thus can be applicable also for future network architectures.

Replay in Deep Learning: Current Approaches and Missing Biological Elements

“…Similarly, replay has been used for continual language learning (de Masson d' Autume et al, 2019). Replay has also been used to perform continual semantic segmentation of medical images (Ozdemir et al, 2018;Ozdemir and Goksel, 2019), remote sensing data (Tasar et al, 2019;Wu et al, 2019b), and on standard computer vision benchmarks (Cermelli et al, 2020). In (Acharya et al, 2020;Liu et al, 2020a), replay is used to mitigate forgetting for an continual object detection approach.…”

Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation

“…Continual Semantic segmentation: Despite enormous progress in the two aforementioned areas respectively, segmentation algorithms are mostly used in an offline setting, while continual learning methods generally focus on image classification. Recent works extend existing continual learning methods [57], [60] for specialized applications [11], [12], [13] and general semantic segmentation [14]. The latter considers that the previously learned categories are properly annotated in the images of the new dataset.…”

“…Idealy, one would wish to regularly expand a dataset, only adding and labelling new classes and updating the model in accordance. This setup, referred here as Continual Semantic Segmentation (CSS), has emerged very recently for specialized applications [11], [12], [13] before being proposed for general segmentation datasets [14], [15], [16].…”

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

“…All these approaches, however, require training data and segmentation maps to be available at once (i.e., joint setting) and they experience catastrophic forgetting if new tasks (e.g., new classes to learn) are made available sequentially [42]. Hence, it emerged the need for continual approaches specifically targeted to solve the semantic segmentation task [47,58,42,43,33,4]. Earlier works focus on the continual semantic segmentation problem in specific scenarios, e.g., in medical imaging [47] or remote sensing [58], extending standard image-level classification methods.…”

“…Hence, it emerged the need for continual approaches specifically targeted to solve the semantic segmentation task [47,58,42,43,33,4]. Earlier works focus on the continual semantic segmentation problem in specific scenarios, e.g., in medical imaging [47] or remote sensing [58], extending standard image-level classification methods. More recently, standard semantic segmentation datasets and targeted methods have been proposed.…”