SFNet: Learning Object-Aware Semantic Correspondence

Abstract

We address the problem of semantic correspondence, that is, establishing a dense flow field between images depicting different instances of the same object or scene category. We propose to use images annotated with binary foreground masks and subjected to synthetic geometric deformations to train a convolutional neural network (CNN) for this task. Using these masks as part of the supervisory signal offers a good compromise between semantic flow methods, where the amount of training data is limited by the cost of manually selecting point correspondences, and semantic alignment ones, where the regression of a single global geometric transformation between images may be sensitive to image-specific details such as background clutter. We propose a new CNN architecture, dubbed SFNet, which implements this idea. It leverages a new and differentiable version of the argmax function for end-to-end training, with a loss that combines mask and flow consistency with smoothness terms. Experimental results demonstrate the effectiveness of our approach, which significantly outperforms the state of the art on standard benchmarks.

A Novel Post-Processing Method Based on a Weighted Composite Filter for Enhancing Semantic Segmentation Results

“…Argmax . The argmax function [ 38 ] of each pixel-bit depth vector is used to decompose the predicted values into segmentation masks and to get the enhancement result. The enhancement method based on WCF is then employed to improve the segmentation result and better capture the object boundaries.…”

Semantic Matching Based on Semantic Segmentation and Neighborhood Consensus

“…In other words, considering context helps to improve the performance of semantic matching. To this end, a direct way is to explicitly add neighborhood continuity constraints to loss, such as smoothness and geometric consistency constraints [34,35]. Another strategy is to consider the spatial context when extracting semantic features so that the features can perceive local information.…”

“…Instead of using the foreground-mask correspondences as supervision signals [34], our method uses keypoint labels since they have pixel-level ground-truth matches. This stronger supervision signal can guide the network to estimate the matching field between images.…”

Boosting Real-Time Driving Scene Parsing With Shared Semantics

“…Wang et al [20] used an unsupervised method to learn feature representations for identifying correspondences across frames. Lee et al [21] attempted to derive semantic correspondences by objectaware losses. Compared with pixel-level sharing, featurelevel sharing is learned by an end-to-end process, and it is thus difficult to directly evaluate its performance.…”