Learning End-to-End Scene Flow by Distilling Single Tasks Knowledge

Abstract

Scene flow is a challenging task aimed at jointly estimating the 3D structure and motion of the sensed environment. Although deep learning solutions achieve outstanding performance in terms of accuracy, these approaches divide the whole problem into standalone tasks (stereo and optical flow) addressing them with independent networks. Such a strategy dramatically increases the complexity of the training procedure and requires power-hungry GPUs to infer scene flow barely at 1 FPS. Conversely, we propose DWARF, a novel and lightweight architecture able to infer full scene flow jointly reasoning about depth and optical flow easily and elegantly trainable end-to-end from scratch. Moreover, since ground truth images for full scene flow are scarce, we propose to leverage on the knowledge learned by networks specialized in stereo or flow, for which much more data are available, to distill proxy annotations. Exhaustive experiments show that i) DWARF runs at about 10 FPS on a single high-end GPU and about 1 FPS on NVIDIA Jetson TX2 embedded at KITTI resolution, with moderate drop in accuracy compared to 10× deeper models, ii) learning from many distilled samples is more effective than from the few, annotated ones available.

Consistency Guided Scene Flow Estimation

“…As shown in table 1, the refinement model improves over the feedforward baseline in all three metrics, thus supporting the effectiveness of the consistency guided refinement process. Our model also outperforms DWARF [1] by a large margin. Again, this shows the benefit of geometrically modelling the consistency of scene flow outputs.…”

“…To leverage correlation between tasks, Jiang et al [15] propose an encoder architecture shared among the tasks of disparity, optical flow, and segmentation. Similarly, Aleotti et al [1] propose a lightweight architecture to share information between tasks. Complementary to previous work, our main objective is to improve generalization of the scene flow by means of additional constraints, self-supervised losses, and learnt refinement schemes.…”

“…We use the results produced by the feedforward module without a refinement network as baseline. To facilitate the comparison with the state-of-the-art, we test against a recent model DWARF [1] in the same experimental setting. As shown in table 1, the refinement model improves over the feedforward baseline in all three metrics, thus supporting the effectiveness of the consistency guided refinement process.…”

“…Many end-to-end models powered by deep neural network have emerged for the scene flow estimation [1,15] and its components [5,13,29], and they showed promise in benchmarks. However, training such models relies on the availability of labeled data.…”

Self-Supervised Monocular Scene Flow Estimation

“…The benefit of our split decoder is that competitive accuracy is achieved more stably and in fewer training iterations (at 56% of the full training schedule), with a lighter network (∼ 10% fewer parameters). 1 Figure 2. Decoder configuration: (a) A single joint decoder [23], (b) removing the context network, and (c) our split decoder design.…”