Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding

Abstract

Learning to estimate 3D geometry in a single image by watching unlabeled videos via deep convolutional network has made significant process recently. Current state-of-the-art (SOTA) methods, are based on the learning framework of rigid structure-from-motion, where only 3D camera ego motion is modeled for geometry estimation. However, moving objects also exist in many videos, e.g. moving cars in a street scene. In this paper, we tackle such motion by additionally incorporating per-pixel 3D object motion into the learning framework, which provides holistic 3D scene flow understanding and helps single image geometry estimation. Specifically, given two consecutive frames from a video, we adopt a motion network to predict their relative 3D camera pose and a segmentation mask distinguishing moving objects and rigid background. An optical flow network is used to estimate dense 2D per-pixel correspondence. A single image depth network predicts depth maps for both images. The four types of information, i.e. 2D flow, camera pose, segment mask and depth maps, are integrated into a differentiable holistic 3D motion parser (HMP), where per-pixel 3D motion for rigid background and moving objects are recovered. We design various losses w.r.t. the two types of 3D motions for training the depth and motion networks, yielding further error reduction for estimated geometry. Finally, in order to solve the 3D motion confusion from monocular videos, we combine stereo images into joint training. Experiments on KITTI 2015 dataset show that our estimated geometry, 3D motion and moving object masks, not only are constrained to be consistent, but also significantly outperforms other SOTA algorithms, demonstrating the benefits of our approach.

Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes

“…An estimate is considered as an error if it exceeds We propose four categories of state-of-the-art monocular baseline methods. In the first category are the multi-task networks, GeoNet [73], DF-Net [77] and EveryPixel [70]. These CNNs are trained in an unsupervised manner and are able to provide single-view depth estimates for both images and optical flow estimates.…”

“…For the GeoNet and DF-Net, their published code and models are used. The results of the EveryPixel approach are stated in their paper [70] (D2 metric is excluded as it seems to be inconsistent). As a second category, single-view depth estimation ('LRC [19]' or 'DORN [14]') and optical flow estimation ('MirrorFlow [30] and 'HD 3 -F [72]' used parts of the dataset for training, these methods are disregarded for ranking.…”

Progressive Fusion for Unsupervised Binocular Depth Estimation Using Cycled Networks

“…Conversely, in this work, we focus on the binocular stereo depth estimation scenario where both images are available at test time. Other works considered jointly learning the scene depth and the ego-motion in monocular videos without using groundtruth data [13], [31], [32], [33], [34]. These works demonstrated that by integrating temporal information and considering multiple consecutive frames better estimates can be obtained.…”

Learning Residual Flow as Dynamic Motion from Stereo Videos

“…With the rapid development of deep convolutional neural networks (CNNs), the most recent approaches [14], [15], [16], [17] address the joint estimation of rigid and non-rigid geometry. They introduce an unsupervised learning framework that estimates depth, optical flow and ego-motion of a camera in a coupled way.…”

“…To tackle this problem, we design an unsupervised learning framework that includes depth, pose and residual flow networks as shown in Figure 1. In contrast to the previous monocular-based approaches [14], [15], [16], [17], we introduce a stereo-based motion and residual flow learning module to handle non-rigid cases caused by dynamic objects in an unsupervised manner. Employing a stereo-based system enables to solve the ambiguity of 3D object motion in the monocular setting, and to obtain reliable depth estimates that can facilitate the other tasks.…”