End-to-End learning of cost-volume aggregation for real-time dense stereo

Kuzmin, Andrey; Mikushin, Dmitry; Lempitsky, Victor

doi:10.1109/mlsp.2017.8168183

Cited by 22 publications

(12 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DispNetC [30] gives very accurate results and this can be seen especially for foreground objects. In terms of accuracy, our solution is as good as DeepCostAggr [25], both methods relying on good object boundary estimates for enhanced reliability. These results prove that our refinement method can compensate the reduced accuracy of cost computation.…”

Section: Results On Kitti 2015 Testing Datasetmentioning

confidence: 90%

See 1 more Smart Citation

Real-Time Semantic Segmentation-Based Stereo Reconstruction

Miclea

Nedevschi

2020

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

In this paper, we propose a novel semantic segmentation-based stereo reconstruction method that can keep up with the accuracy of the state-of-the art approaches while running in real time. The solution follows the classic stereo pipeline, each step in the stereo workflow being enhanced by additional information from semantic segmentation. Therefore, we introduce several improvements to computation, aggregation, and optimization by adapting existing techniques to integrate additional surface information given by each semantic class. For the cost computation and optimization steps, we propose new genetic algorithms that can incrementally adjust the parameters for better solutions. Furthermore, we propose a new postprocessing edge-aware filtering technique relying on an improved convolutional neural network (CNN) architecture for disparity refinement. We obtain the competitive results at 30 frames/s, including segmentation.

show abstract

Section: Results On Kitti 2015 Testing Datasetmentioning

confidence: 90%

“…This approach controls the aggregation window by 2 parameters corresponding to locality and intensity similarity. A different technique is proposed by [25] in which a residual neural network is responsible for edge detection. Edges are detected at different scales and are later used for setting the aggregation window boundaries.…”

Section: A Classic Taxonomymentioning

confidence: 99%

Real-Time Semantic Segmentation-Based Stereo Reconstruction

Miclea

Nedevschi

2020

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

show abstract

“…Instead, we propose a method that uses only a single, raster-friendly minimisation, but one that incorporates information from four directions at once. Our approach compares favourably with the two state-ofthe-art GPU-based methods [15], [24] that can process the KITTI dataset in real time, achieving similar levels of accuracy whilst reducing the power consumption by two orders of magnitude. Moreover, in comparison to other FPGA-based methods on the Middlebury dataset, we achieve comparable accuracy either at a much higher frame-rate (c.f.…”

Section: Discussionmentioning

confidence: 96%

“…On KITTI, we compare our approach to the only two published approaches from the benchmark that are able to achieve state-of-the-art performance in real time [15], [24], both of which require a powerful GPU (an Nvidia GTX Titan X) to run. Since, unlike these approaches, our approach does not naturally produce disparities for every single pixel in the image, we interpolate as specified by the KITTI evaluation protocol in order to make our results comparable with theirs.…”

Section: Methodsmentioning

confidence: 99%

R3SGM: Real-Time Raster-Respecting Semi-Global Matching for Power-Constrained Systems

Rahnama¹,

Cavallari²,

Golodetz³

et al. 2018

2018 International Conference on Field-Programmable Technology (FPT)

View full text Add to dashboard Cite

Stereo depth estimation is used for many computer vision applications. Though many popular methods strive solely for depth quality, for real-time mobile applications (e.g. prosthetic glasses or micro-UAVs), speed and power efficiency are equally, if not more, important. Many real-world systems rely on Semi-Global Matching (SGM) to achieve a good accuracy vs. speed balance, but power efficiency is hard to achieve with conventional hardware, making the use of embedded devices such as FPGAs attractive for low-power applications. However, the full SGM algorithm is ill-suited to deployment on FPGAs, and so most FPGA variants of it are partial, at the expense of accuracy. In a non-FPGA context, the accuracy of SGM has been improved by More Global Matching (MGM), which also helps tackle the streaking artifacts that afflict SGM. In this paper, we propose a novel, resource-efficient method that is inspired by MGM's techniques for improving depth quality, but which can be implemented to run in real time on a low-power FPGA. Through evaluation on multiple datasets (KITTI and Middlebury), we show that in comparison to other real-time capable stereo approaches, we can achieve a state-of-the-art balance between accuracy, power efficiency and speed, making our approach highly desirable for use in real-time systems with limited power.

show abstract

“…The goal of our work is accurate but real-time stereo estimation, which we achieve through the use of traditional matching costs and 2D (instead of 3D) convolution layers. In this context, it is useful to discuss the work of Kuzmin et al [17], who also use traditional matching costs as well as a largely traditional pipeline for aggregation, using a learned deep network to control the parameters of this aggregation in different regions. This allows them to achieve a low runtime of 0.034 seconds (i.e., 29.4 frames per second) but with worse accuracy.…”

Section: Background and Related Workmentioning

confidence: 99%

Fast Deep Stereo with 2D Convolutional Processing of Cost Signatures

Yee¹,

Chakrabarti²

2019

Preprint

View full text Add to dashboard Cite

Modern neural network-based algorithms are able to produce highly accurate depth estimates from stereo image pairs, nearly matching the reliability of measurements from more expensive depth sensors. However, this accuracy comes with a higher computational cost since these methods use network architectures designed to compute and process matching scores across all candidate matches at all locations, with floating point computations repeated across a match volume with dimensions corresponding to both space and disparity. This leads to longer running times to process each image pair, making them impractical for real-time use in robots and autonomous vehicles. We propose a new stereo algorithm that employs a significantly more efficient network architecture. Our method builds an initial match cost volume using traditional matching costs that are fast to compute, and trains a network to estimate disparity from this volume. Crucially, our network only employs per-pixel and two-dimensional convolution operations: to summarize the match information at each location as a low-dimensional feature vector, and to spatially process these "cost-signature" features to produce a dense disparity map. Experimental results on the KITTI benchmark show that our method delivers competitive accuracy at significantly higher speeds-running at 48 frames per second on a modern GPU.

show abstract

End-to-End learning of cost-volume aggregation for real-time dense stereo

Cited by 22 publications

References 34 publications

Real-Time Semantic Segmentation-Based Stereo Reconstruction

Real-Time Semantic Segmentation-Based Stereo Reconstruction

R3SGM: Real-Time Raster-Respecting Semi-Global Matching for Power-Constrained Systems

Fast Deep Stereo with 2D Convolutional Processing of Cost Signatures

Contact Info

Product

Resources

About