Path-Invariant Map Networks

Abstract

Optimizing a network of maps among a collection of objects/domains (or map synchronization) is a central problem across computer vision and many other relevant fields. Compared to optimizing pairwise maps in isolation, the benefit of map synchronization is that there are natural constraints among a map network that can improve the quality of individual maps. While such self-supervision constraints are well-understood for undirected map networks (e.g., the cycle-consistency constraint), they are underexplored for directed map networks, which naturally arise when maps are given by parametric maps (e.g., a feedforward neural network). In this paper, we study a natural self-supervision constraint for directed map networks called path-invariance, which enforces that composite maps along different paths between a fixed pair of source and target domains are identical. We introduce path-invariance bases for efficient encoding of the path-invariance constraint and present an algorithm that outputs a path-variance basis with polynomial time and space complexities. We demonstrate the effectiveness of our approach on optimizing object correspondences, estimating dense image maps via neural networks, and semantic segmentation of 3D scenes via map networks of diverse 3D representations. In particular, for 3D semantic segmentation our approach only requires 8% labeled data from ScanNet to achieve the same performance as training a single 3D segmentation network with 30% to 100% labeled data. * Xiaowei Zhou is affiliated with the StateKey Lab of CAD&CG and the ZJU-SenseTime Joint Lab of 3D Vision. †

HybridPose: 6D Object Pose Estimation under Hybrid Representations

“…In the future, we plan to extend HybridPose to include more intermediate representations such as shape primitives, normals, and planar faces. Another possible direction is to enforce consistency across different representations in a similar way to [46] as a self-supervision loss in network training.…”

Quantum Permutation Synchronization

“…The notion of cycle consistency for directed graphs is known as path invariance [108] and differs from cycle consistency as shown in Fig. 2.…”

“…This procedure is a fundamental piece of most state-of-the-art multi-view both authors contributed equally to this work reconstruction and multi-shape analysis pipelines [86,23,25] because it heavy-lifts the global constraint satisfaction while respecting the geometry of the parameters. In fact, most of the multiview-consistent inference problems can be expressed as some form of a synchronization [108,15]. In this paper, our focus is permutation synchronization, where the edges of the graph are labeled by permutation matrices denoting the correspondences either between two 2D images or two 3D shapes.…”

Predicting What You Already Know Helps: Provable Self-Supervised Learning

“…A popular approach for the domain of text is based on language modeling where models like BERT and GPT create auxiliary tasks for next word predictions [9,39]. The natural ordering or topology of data is also exploited in video-based [51,35,13], graph-based [52,25] or map-based [55] self-supervised learning. For instance, the pretext task is to determine the correct temporal order for video frames as in [35].…”