Fast Deep Stereo with 2D Convolutional Processing of Cost Signatures

Abstract

Modern neural network-based algorithms are able to produce highly accurate depth estimates from stereo image pairs, nearly matching the reliability of measurements from more expensive depth sensors. However, this accuracy comes with a higher computational cost since these methods use network architectures designed to compute and process matching scores across all candidate matches at all locations, with floating point computations repeated across a match volume with dimensions corresponding to both space and disparity. This leads to longer running times to process each image pair, making them impractical for real-time use in robots and autonomous vehicles. We propose a new stereo algorithm that employs a significantly more efficient network architecture. Our method builds an initial match cost volume using traditional matching costs that are fast to compute, and trains a network to estimate disparity from this volume. Crucially, our network only employs per-pixel and two-dimensional convolution operations: to summarize the match information at each location as a low-dimensional feature vector, and to spatially process these "cost-signature" features to produce a dense disparity map. Experimental results on the KITTI benchmark show that our method delivers competitive accuracy at significantly higher speeds-running at 48 frames per second on a modern GPU.

DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion

“…On the one hand, methods such as MVSNet [50] and DPSNet [20] build 4D feature volumes [24], and regularize the feature volumes by employing 3D convolutions, a process that delivers high accuracy, but also is demanding in terms of memory and compute. On the other hand, MVDepthNet [48] and [51], directly generate 3D volumes by computing traditional cost measures on image features or RGB values. This allows the network architecture to be based on 2D convolutions, which are much faster than the 3D counterpart, and better suited for real-time applications.…”

Improving Deep Stereo Network Generalization with Geometric Priors

“…However, PSM-Net still uses 3D convolutions and cannot process high-resolution images because of GPU memory constraints. Later meth-ods reduce the computational overhead and/or aim for highresolution images [11,27,22,24,20,28]. In particular, HSM-Net [27] estimates disparity in a coarse-to-fine manner, and uses novel data augmentation methods to achieve state-of-the-art in mean absolute error (MAE) in the highresolution Middlebury dataset while running with low latency.…”