Demystifying Unsupervised Semantic Correspondence Estimation

Abstract

We explore semantic correspondence estimation through the lens of unsupervised learning. We thoroughly evaluate several recently proposed unsupervised methods across multiple challenging datasets using a standardized evaluation protocol where we vary factors such as the backbone architecture, the pre-training strategy, and the pre-training and finetuning datasets. To better understand the failure modes of these methods, and in order to provide a clearer path for improvement, we provide a new diagnostic framework along with a new performance metric that is better suited to the semantic matching task. Finally, we introduce a new unsupervised correspondence approach which utilizes the strength of pre-trained features while encouraging better matches during training. This results in significantly better matching performance compared to current state-of-the-art methods.

Learning Universal Semantic Correspondences with No Supervision and Automatic Data Curation

“…While its unsupervised objective clusters semantically similar features, here we do the opposite and learn features that lead to unique corre-spondences. Aygun et al [2] use an equivariant representation learning approach, inspired by [24,53,54]. Similar to our work, the method learns to project self-supervised ViT features to a semantic correspondence space.…”

“…3) to perform best. The La architecture, similar to [2], cannot account for the appearance of the target features by design. The LLA architecture, can do so but learns a degenerate solution: the model can satisfy Eq.…”

“…They have sparse keypoint labels, and contain images with varying appearances, poses, and backgrounds, making them particularly challenging. We test on 10,000 random pairs (the same as in [2]). Spair-71K is the most challenging dataset of the four, and the only one collected for the task of semantic correspondences.…”

“…All images are resized to 224×224 and the ViT uses 8 × 8 patches with stride 8 are used as input. We use the 384-dimensional key features from the 9th layer, similarly to prior work [1,2]. The temperature for the contrastive distillation loss L d in Eq.…”

ASIC: Aligning Sparse in-the-wild Image Collections

“…To overcome the lack of large datasets with ground-truth correspondence, recent work seeks to combine the idea of distilling deep features from a network trained with self-supervision on large-scale image datasets. Some of these works optimize for proxy losses computed with known transformations [4,33,40,58,63,70,73,74,77]. Like these methods, we also train our network to be equivariant to synthetic geometric transformations.…”