Online supervised attention-based recurrent depth estimation from monocular video

Abstract

Autonomous driving highly depends on depth information for safe driving. Recently, major improvements have been taken towards improving both supervised and self-supervised methods for depth reconstruction. However, most of the current approaches focus on single frame depth estimation, where quality limit is hard to beat due to limitations of supervised learning of deep neural networks in general. One of the way to improve quality of existing methods is to utilize temporal information from frame sequences. In this paper, we study intelligent ways of integrating recurrent block in common supervised depth estimation pipeline. We propose a novel method, which takes advantage of the convolutional gated recurrent unit (convGRU) and convolutional long short-term memory (convLSTM). We compare use of convGRU and convLSTM blocks and determine the best model for real-time depth estimation task. We carefully study training strategy and provide new deep neural networks architectures for the task of depth estimation from monocular video using information from past frames based on attention mechanism. We demonstrate the efficiency of exploiting temporal information by comparing our best recurrent method with existing image-based and video-based solutions for monocular depth reconstruction.

Self-supervised recurrent depth estimation with attention mechanisms

“…Over the past few years, some studies have used an alternative recurrent mechanism called ConvGRU ( Zhang et al., 2019 ). It was first introduced for the video representation task ( Ballas et al., 2015 ) and has proven to be quite effective in subsequent studies ( Siam et al., 2016 ; Maslov & Makarov, 2020a ). ConvGRU has a simpler architecture than ConvLSTM while preserving the same gated structure.…”

“…ConvGRU has a simpler architecture than ConvLSTM while preserving the same gated structure. In this work, we use ConvGRU cells since it has fewer parameters, and it has been proven to be effective in capturing temporal information for online depth estimation ( Maslov & Makarov, 2020a ).…”

“…Our main idea of using convGRU unit is similar to Maslov & Makarov (2020a) , in which authors used combinations of convolutions and recurrent blocks for temporal information aggregation for supervised depth estimation. We believe that in self-supervised setting temporal consistency is even more essential for robust depth estimation in both, single image and sequential settings.…”

“…Therefore, the impact of the hidden state becomes very important. We follow Maslov & Makarov (2020a) and Patil et al (2020) and use two training stages. On the first stage, we train a model for hidden states considering them as parameters, at the second stage we use learned hidden states as initial states for every sequence.…”

“…Therefore, a challenge arises to build a lightweight model that is able to process frame sequences in real-time with reasonable quality. Recently, Maslov & Makarov (2020a) proposed an architecture that appears to be suitable for real-time robotic applications. The model utilizes temporal information from image sequences with the help of a convolutional gated recurrent unit (ConvGRU) and convolutional long short-term memory (ConvLSTM).…”