Renovate Yourself: Calibrating Feature Representation of Misclassified Pixels for Semantic Segmentation

Abstract

Existing image semantic segmentation methods favor learning consistent representations by extracting long-range contextual features with the attention, multi-scale, or graph aggregation strategies. These methods usually treat the misclassified and correctly classified pixels equally, hence misleading the optimization process and causing inconsistent intra-class pixel feature representations in the embedding space during learning. In this paper, we propose the auxiliary representation calibration head (RCH), which consists of the image decoupling, prototype clustering, error calibration modules and a metric loss function, to calibrate these error-prone feature representations for better intra-class consistency and segmentation performance. RCH could be incorporated into the hidden layers, trained together with the segmentation networks, and decoupled in the inference stage without additional parameters. Experimental results show that our method could significantly boost the performance of current segmentation methods on multiple datasets (e.g., we outperform the original HRNet and OCRNet by 1.1% and 0.9% mIoU on the Cityscapes test set). Codes are available at https://github.com/VipaiLab/RCH.

Learning Discriminative Representations for Skeleton Based Action Recognition

“…He et al [3,4,12] realize contrastive learning in a more efficient way using a momentum encoder and a dynamic queue. Wang et al [33] define the positive and negative samples in a supervised manner. They design a metric function loss to calibrate these misclassified feature representations for better intra-class consistency and segmentation performance.…”

“…2 shows, we conduct the feature refinement in a manner of contrastive learning. The idea is inspired by [33]. Each sample will be refined by both its ground truth actions and other ambiguous actions.…”