On advantages of Mask-level Recognition for Open-set Segmentation in the Wild

Abstract

Most dense recognition methods bring a separate decision in each particular pixel. This approach still delivers competitive performance in usual closed-set setups with small taxonomies. However, important applications in the wild typically require strong open-set performance and large numbers of known classes. We show that these two demanding setups greatly benefit from mask-level predictions, even in the case of non-finetuned baseline models. Moreover, we propose an alternative formulation of dense recognition uncertainty that effectively reduces false positive responses at semantic borders. The proposed formulation produces a further improvement over a very strong baseline and sets the new state of the art in dense anomaly detection without training on negative data. Our contributions also lead to a performance improvement in a recent open-set panoptic setup. In-depth experiments confirm that our approach succeeds due to implicit aggregation of pixellevel cues into mask-level predictions.

CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen

“…However, these methods result in noisy predictions due to a lack of structured knowledge. Current techniques, such as those in (Nayal et al 2023;Grcić, Šarić, and Šegvić 2023), tackle this issue using the region-based framework, Mask2Former (M2F) (Cheng et al 2021). However, to achieve optimal performance, they necessitate OOD data and complete train the model from scratch.…”