Unsupervised Learning of Camera Pose with Compositional Re-estimation

On the Coupling of Depth and Egomotion Networks for Self-Supervised Structure from Motion

“…First, we survey the field and categorize the approaches to coupling described in the literature (Figure 1 and Table 1). We find that most systems rely on indirect coupling of depth and egomotion via the self-supervised reconstruction loss; others incorporate direct coupling by treating one prediction as a function of the other [42,23]; and lastly, a final set of methods [28,14] incorporate a form of direct coupling we call feedback coupling to iteratively refine predictions based on successive forward passes.…”

“…Baseline [50,13,15] ---PFT [49,5,24,27,34,20] --Ambrus et al [1] -D→E -ManyDepth [44] E→D -Nabavi et al [28] -D→E DRO [14] -D E Ours D→E Table 1: Summarizing the degree of coupling between depth and egomotion that are retained at inference time. Ours is the only method that uses all three forms of coupling.…”

“…Building on this taxonomy, we present a novel network structure that ensures the depth and egomotion network predictions are tightly coupled at both training and inference time by incorporating all three coupling strategies. Our approach leverages two specific methods -namely, test-time optimization [49,5,44,24,27,34,20] for parameter fine-tuning (PFT), and iterative view synthesis [28] to recursively update the egomotion network input with the most recent synthesized view. Through extensive experiments, we demonstrate that our approach promotes consistency between the depth and egomotion predictions at test time, improves generalization on new data, and leads to state-of-the-art accuracy on indoor and outdoor depth and egomotion evaluation benchmarks.…”

“…We identify two existing self-supervised feedback coupling methods in the literature. Nabavi et al [28] use a photometric cost map consisting of the target view, and a synthetic target view (generated from a nearby source view); optimal depth and egomotion predictions maximally align the synthesized view with the target view, which effectively minimizes the cost map. The authors of Deep Recurrent Optimizer (DRO) [14] adopt a similar approach, but replace the photometric cost map with a feature-metric one; further, they train separate (iterative) depth and egomotion "optimizers" that take as input the feature-metric cost map.…”

“…which is a weighted combination of the L 1 loss and a structural similarity (SSIM) loss [43]. A spatial transformer [18] is used to reconstruct I s from I s by populating each target image pixel coordinate u with the value from its corresponding location in the source image: We use feedback coupling through iterative view synthesis [28] to produce an egomotion prediction that is notably is a function of depth. At inference time, the depth network weights are updated via gradient descent to further minimize the sample-wise loss, and as a result of our unique coupling scheme, both depth and egomotion are improved.…”