Sylph: A Hypernetwork Framework for Incremental Few-shot Object Detection

Abstract

Few-shot object detection(FSOD) aims to design methods to adapt object detectors efficiently with only few annotated samples. Fine-tuning has been shown to be an effective and practical approach. However, previous works often take the classical base-novel two stage fine-tuning procedure but ignore the implicit stability-plasticity contradiction among different modules. Specifically, the random re-initialized classifiers need more plasticity to adapt to novel samples. The other modules inheriting pre-trained weights demand more stability to reserve their class-agnostic knowledge. Regular fine-tuning which couples the optimization of these two parts hurts the model generalization in FSOD scenarios. In this paper, we find that this problem is prominent in the end-to-end object detector Sparse R-CNN for its multi-classifier cascaded architecture. We propose to mitigate this contradiction by a new three-stage fine-tuning procedure by introducing an addtional plasticity classifier fine-tuning(PCF) stage. We further design the multi-source ensemble(ME) technique to enhance the generalization of the model in the final finetuning stage. Extensive experiments verify that our method is effective in regularizing Sparse R-CNN, outperforming previous methods in the FSOD benchmark.

Incremental Few-Shot Object Detection via Simple Fine-Tuning Approach

“…Implementation details 1) Datasets and metric: We evaluated iTFA on PASCAL VOC 2012 [16], COCO [17], and LVIS v1 [18] datasets. The evaluation metric and the data split strategy were followed from [6], [15], [20] for a fair comparison. For COCO, 60 categories disjoint from PASCAL VOC are considered base classes; the other 20 classes are considered novel classes.…”

“…Due to this assumption, catastrophic forgetting occurs; the model adds novel classes, then performance degradation occurs to the base classes. Like the FSOD workflow, prior works [15], [20] have developed meta-learning-based methods to solve these problems. ONCE [15] presented a class code generator to generate classification heads for novel classes.…”

“…ONCE [15] presented a class code generator to generate classification heads for novel classes. Sylph [20] developed a code predictor head and a code process module to solve iFSD without additional training and forgetting the base classes. Meta-learning-based methods can be directly adapted to novel classes; however, they have low accuracy.…”

“…TFA [6] is a fine-tuning-based approach for FSOD. ONCE [15] and Sylph [20] are the meta-learning-based approaches for iFSD; they use a one-stage object detector due to its superior generalization performance. iMTFA [31] is designed for incremental few-shot semantic segmentation based on Mask R-CNN [4]; it creates novel classifier weights through weight imprinting.…”

Continual Few-Shot Learning Using HyperTransformers

“…Incremental few-shot learning A related, but distinct area of research is incremental few-shot learning (Gidaris & Komodakis, 2018;Ren et al, 2019;Perez-Rua et al, 2020;Chen & Lee, 2020;Wang et al, 2021b;Shi et al, 2021;Mazumder et al, 2021;Lee et al, 2021;Yin et al, 2022). There, the goal is to adapt a few-shot task to an existing base classifier trained on a large dataset, without forgetting the original data.…”