Exploiting Temporal Consistency for Real-Time Video Depth Estimation

Abstract

Accuracy of depth estimation from static images has been significantly improved recently, by exploiting hierarchical features from deep convolutional neural networks (CNNs). Compared with static images, vast information exists among video frames and can be exploited to improve the depth estimation performance. In this work, we focus on exploring temporal information from monocular videos for depth estimation. Specifically, we take the advantage of convolutional long short-term memory (CLSTM) and propose a novel spatial-temporal CSLTM (ST-CLSTM) structure. Our ST-CLSTM structure can capture not only the spatial features but also the temporal correlations/consistency among consecutive video frames with negligible increase in computational cost. Additionally, in order to maintain the temporal consistency among the estimated depth frames, we apply the generative adversarial learning scheme and design a temporal consistency loss. The temporal consistency loss is combined with the spatial loss to update the model in an end-to-end fashion. By taking advantage of the temporal information, we build a video depth estimation framework that runs in real-time and generates visually pleasant results. Moreover, our approach is flexible and can be generalized to most existing depth estimation frameworks. Code is available at:

Don’t Forget The Past: Recurrent Depth Estimation from Monocular Video

“…Our method is designed for online depth estimation in videos. Similar idea of online estimation is proposed in recent works [30], [31], where they use ConvLSTM but in supervised framework. There are also earlier methods for offline depth estimation from videos [32], in which local motion cues and optical flow are used to produce temporally consistent depth maps.…”

“…7. The quantitative metrices defined by [30] are not suitable for Datasets with sparse ground truth. We define our evaluation metric, Absolute Relative Temporal Error(ARTE), as follows: .…”

Temporal Denoising Mask Synthesis Network for Learning Blind Video Temporal Consistency

“…Meanwhile, they not only compute the short-term temporal loss, but also calculate the long-time temporal loss by the frame pairs which are sampled from the output sequence. Zhang et al [48] propose a ST-CLSTM structure which extracts both spatial features and temporal correlations to retain the temporal consistency. In this paper, we teach our network to learn the temporal consistency from the original videos.…”

Multi-Scale Dilated Convolution Network Based Depth Estimation in Intelligent Transportation Systems

“…The resultant combined global and local feature clues are then used to form the cost volume for the following reliable disparity estimation. Recently, more and more works [49]- [53] concentrate on monocular depth estimation field. Godard et al [49] used self-supervised learning to training models in order to resolve the problem that per-pixel ground-truth depth data is challenging to acquire at scale.…”