CoMoDA: Continuous Monocular Depth Adaptation Using Past Experiences

Abstract

While ground truth depth data remains hard to obtain, self-supervised monocular depth estimation methods enjoy growing attention. Much research in this area aims at improving loss functions or network architectures. Most works, however, do not leverage self-supervision to its full potential. They stick to the standard closed world train-test pipeline, assuming the network parameters to be fixed after the training is finished. Such an assumption does not allow to adapt to new scenes, whereas with self-supervision this becomes possible without extra annotations.In this paper, we propose a novel self-supervised Continuous Monocular Depth Adaptation method (CoMoDA), which adapts the pretrained model on a test video on the fly. As opposed to existing test-time refinement methods that use isolated frame triplets, we opt for continuous adaptation, making use of the previous experience from the same scene. We additionally augment the proposed procedure with the experience from the distant past, preventing the model from overfitting and thus forgetting already learnt information.We demonstrate that our method can be used for both intra-and cross-dataset adaptation. By adapting the model from train to test set of the Eigen split of KITTI, we achieve state-of-the-art depth estimation performance and surpass all existing methods using standard architectures. We also show that our method runs 15 times faster than existing test-time refinement methods.

On the Coupling of Depth and Egomotion Networks for Self-Supervised Structure from Motion

“…Baseline [50,13,15] ---PFT [49,5,24,27,34,20] --Ambrus et al [1] -D→E -ManyDepth [44] E→D -Nabavi et al [28] -D→E DRO [14] -D E Ours D→E Table 1: Summarizing the degree of coupling between depth and egomotion that are retained at inference time. Ours is the only method that uses all three forms of coupling.…”

“…Building on this taxonomy, we present a novel network structure that ensures the depth and egomotion network predictions are tightly coupled at both training and inference time by incorporating all three coupling strategies. Our approach leverages two specific methods -namely, test-time optimization [49,5,44,24,27,34,20] for parameter fine-tuning (PFT), and iterative view synthesis [28] to recursively update the egomotion network input with the most recent synthesized view. Through extensive experiments, we demonstrate that our approach promotes consistency between the depth and egomotion predictions at test time, improves generalization on new data, and leads to state-of-the-art accuracy on indoor and outdoor depth and egomotion evaluation benchmarks.…”

“…Although this form of coupling is sufficient to jointly learn depth and egomotion, the major drawback is that the networks become decoupled at test time, which prevents contextual information (such as scale) from being passed from one network to the other. Recently, however, it has been proposed thatowing to its self-supervised nature -the reconstruction loss can be retained at inference time, and be further minimized by a gradient-descent-based optimizer [49,5,44,24,27,34,20]. In doing so, the indirect link between networks (via gradient flow from the loss function) is preserved at inference time.…”

Continual SLAM: Beyond Lifelong Simultaneous Localization and Mapping through Continual Learning

“…Most CL approaches employ one of three strategies [18]: 1) experience replay including rehearsal and generative replay, 2) regularization, and 3) architectural approaches. Rehearsal refers to saving raw data samples of previous tasks, thereby handling the memory aspect, and re-using them during adaptation to new tasks, e.g., the replay buffer in CoMoDA [19]. To minimize the required size of the memory, the most representative samples should be carefully chosen or replaced by more abstract knowledge representations, e.g., flashcards [20].…”

The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth

“…Test-time refinement approaches adapt monocular methods to use sequence information at test time e.g. [5,9,59,62,72,51]. As self-supervised training does not require any ground truth depth supervision, the same losses used during training can be applied to the test frames to update the network's parameters.…”