Real-Time Dense Stereo Matching With ELAS on FPGA-Accelerated Embedded Devices

Rahnama, Oscar; Frost, Duncan; Mikšík, Ondřej; Torr, Philip H. S.

doi:10.1109/lra.2018.2800786

Cited by 35 publications

(22 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) Redundancy Checking and Disparity Prior Generation: To produce the anchor image (see Figure 5), we further sparsify the support point image from §I-B1 by processing it in raster order and invalidating any pixel whose disparity has already been seen within a window behind and above it. Unlike [28], which for each pixel (x, y) used a window of where K was set to 5, which only encompassed points in the same row or same column as the pixel being processed, here we use a larger window of…”

Section: B Generation Of Priorsmentioning

confidence: 99%

“…the Xilinx ZCU104, which combines both an ARM processor and an FPGA, with shared direct memory access, into a single chip. Recently, several works have explored the deployment of stereo methods on such platforms: both [26] and [19] use the CPU mainly for handling communication and controlling peripherals, while [28] actively leverages the CPU to execute iterative steps that would be infeasible on an FPGA (e.g. Delaunay triangulation).…”

mentioning

confidence: 99%

“…Delaunay triangulation). Similar to [28], we propose to actively use the elaboration capability of the builtin CPU to handle I/O and to execute part of the ELAS pipeline, while deploying all the other elaboration blocks on the FPGA. We show how our pipeline outperforms previously published works by achieving an 8.7% error rate on the challenging KITTI 2015 dataset [29], [30], while still operating with realtime performance and low power consumption.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoC

Rahnama

Cavallari²,

Golodetz³

et al. 2019

IEEE Trans. Circuits Syst. II

Self Cite

View full text Add to dashboard Cite

Obtaining highly accurate depth from stereo images in real time has many applications across computer vision and robotics, but in some contexts, upper bounds on power consumption constrain the feasible hardware to embedded platforms such as FPGAs. Whilst various stereo algorithms have been deployed on these platforms, usually cut down to better match the embedded architecture, certain key parts of the more advanced algorithms, e.g. those that rely on unpredictable access to memory or are highly iterative in nature, are difficult to deploy efficiently on FPGAs, and thus the depth quality that can be achieved is limited. In this paper, we leverage a FPGA-CPU chip to propose a novel, sophisticated, stereo approach that combines the best features of SGM and ELAS-based methods to compute highly accurate dense depth in real time. Our approach achieves an 8.7% error rate on the challenging KITTI 2015 dataset at over 50 FPS, with a power consumption of only 5W.Obtaining information about the 3D structure of a scene is important for many computer vision and robotics applications, e.g. 3D scene reconstruction [1]-[3], camera relocalisation [4]-[6], navigation and obstacle avoidance [7]. Often, this information will be obtained in the form of a depth image, and various options for acquiring such images exist. Passive approaches, which rely only on one or more image sensors, are popular due their low cost, low weight and size, lack of active/moving components, ability to work at longer ranges, deployability in a wider range of operating environments and lack of interference. Among them, binocular stereo relies on a pair of synchronised cameras to acquire the same scene from two different points of view. Given the two frames, a dense and reliable depth map can be computed by finding correspondences between the pixels in the two images [8]. State-of-the-art algorithms for this problem usually rely on costly global image optimisations or on massive convolutional neural networks that involve significant computational costs, making them hard to deploy on resource-limited systems such as embedded devices [9].Two popular solutions offering a good trade-off between speed and accuracy are Semi-Global Matching (SGM) [10] and ELAS [11]. SGM computes initial matching hypotheses by comparing patches around pixels in the left and right images, then approximates a costly image-wide smoothness constraint with the sum of several directional minimizations over the Correspondence: {oscar@robots.ox.ac.uk} O. Rahnama is with the University of Oxford and FiveAI Ltd. T. Joy and P. Torr are with the University of Oxford. A. Tonioni and L. Di Stefano are with the University of Bologna. T. Cavallari, S. Golodetz and S. Walker are with FiveAI Ltd. Work done whilst A. Tonioni was visiting the University of Oxford.disparity range. By contrast, ELAS first identifies a set of sparse but reliable correspondences to provide a coarse approximation of the scene geometry, then uses them to define slanted plane priors that guide the final dense matching stage. ...

show abstract

Section: B Generation Of Priorsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoC

Rahnama

Cavallari²,

Golodetz³

et al. 2019

IEEE Trans. Circuits Syst. II

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, an oversized window not only consumes computational resources but also makes too many errors in matching. Therefore, the size of the window in CT is one of the important keys to determining the performance [ 19 ].…”

Section: Introductionmentioning

confidence: 99%

Improving Census Transform by High-Pass with Haar Wavelet Transform and Edge Detection

Liaw

Huang

et al. 2020

Sensors

View full text Add to dashboard Cite

One of the common methods for measuring distance is to use a camera and image processing algorithm, such as an eye and brain. Mechanical stereo vision uses two cameras to shoot the same object and analyzes the disparity of the stereo vision. One of the most robust methods to calculate disparity is the well-known census transform, which has the problem of conversion window selection. In this paper, three methods are proposed to improve the performance of the census transform. The first one uses a low-pass band of the wavelet to reduce the computation loading and a high-pass band of the wavelet to modify the disparity. The main idea of the second method is the adaptive size selection of the conversion window by edge information. The third proposed method is to apply the adaptive window size to the previous sparse census transform. In the experiments, two indexes, percentage of bad matching pixels (PoBMP) and root mean squared (RMS), are used to evaluate the performance with the known ground truth data. According to the results, the computation required can be reduced by the multiresolution feature of the wavelet transform. The accuracy is also improved with the modified disparity processing. Compared with previous methods, the number of operation points is reduced by the proposed adaptive window size method.

show abstract

“…It has thus proved highly popular in real-world systems, and many FPGA-based approaches have been inspired by it [2], [10], [20], [28], [41], [42]. Other FPGA-based methods have also been presented [34], [36], [37], [46], [50], but, whilst typically faster than those inspired by SGM, they seldom reach the same level of accuracy. However, because the disparities that SGM computes for neighbouring pixels are based on star-shaped sets of input pixels that are mostly disjoint, SGM suffers from streaking in areas in which the data terms in some directions are weak, whilst those in other directions are strong.…”

Section: Introductionmentioning

confidence: 99%

R3SGM: Real-Time Raster-Respecting Semi-Global Matching for Power-Constrained Systems

Rahnama¹,

Cavallari²,

Golodetz³

et al. 2018

2018 International Conference on Field-Programmable Technology (FPT)

Self Cite

View full text Add to dashboard Cite

Stereo depth estimation is used for many computer vision applications. Though many popular methods strive solely for depth quality, for real-time mobile applications (e.g. prosthetic glasses or micro-UAVs), speed and power efficiency are equally, if not more, important. Many real-world systems rely on Semi-Global Matching (SGM) to achieve a good accuracy vs. speed balance, but power efficiency is hard to achieve with conventional hardware, making the use of embedded devices such as FPGAs attractive for low-power applications. However, the full SGM algorithm is ill-suited to deployment on FPGAs, and so most FPGA variants of it are partial, at the expense of accuracy. In a non-FPGA context, the accuracy of SGM has been improved by More Global Matching (MGM), which also helps tackle the streaking artifacts that afflict SGM. In this paper, we propose a novel, resource-efficient method that is inspired by MGM's techniques for improving depth quality, but which can be implemented to run in real time on a low-power FPGA. Through evaluation on multiple datasets (KITTI and Middlebury), we show that in comparison to other real-time capable stereo approaches, we can achieve a state-of-the-art balance between accuracy, power efficiency and speed, making our approach highly desirable for use in real-time systems with limited power.

show abstract

Real-Time Dense Stereo Matching With ELAS on FPGA-Accelerated Embedded Devices

Cited by 35 publications

References 39 publications

Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoC

Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoC

Improving Census Transform by High-Pass with Haar Wavelet Transform and Edge Detection

R3SGM: Real-Time Raster-Respecting Semi-Global Matching for Power-Constrained Systems

Contact Info

Product

Resources

About