Efficient Surface-Aware Semi-Global Matching With Multi-View Plane-Sweep Sampling

Abstract

Online augmentation of an oblique aerial image sequence with structural information is an essential aspect in the process of 3D scene interpretation and analysis. One key aspect in this is the efficient dense image matching and depth estimation. Here, the Semi-Global Matching (SGM) approach has proven to be one of the most widely used algorithms for efficient depth estimation, providing a good trade-off between accuracy and computational complexity. However, SGM only models a first-order smoothness assumption, thus favoring fronto-parallel surfaces. In this work, we present a hierarchical algorithm that allows for efficient depth and normal map estimation together with confidence measures for each estimate. Our algorithm relies on a plane-sweep multi-image matching followed by an extended SGM optimization that allows to incorporate local surface orientations, thus achieving more consistent and accurate estimates in areas made up of slanted surfaces, inherent to oblique aerial imagery. We evaluate numerous configurations of our algorithm on two different datasets using an absolute and relative accuracy measure. In our evaluation, we show that the results of our approach are comparable to the ones achieved by refined Structure-from-Motion (SfM) pipelines, such as COLMAP, which are designed for offline processing. In contrast, however, our approach only considers a confined image bundle of an input sequence, thus allowing to perform an online and incremental computation at 1Hz−2Hz.

Real-Time Dense 3d Reconstruction From Monocular Video Data Captured by Low-Cost Uavs

“…We extend our previous work on efficient depth estimation (Ruf et al, 2019), which is based on a real-time multi-image matching with plane-sweep sampling, by combining it with the stateof-the-art ORB-SLAM2 algorithm (Mur-Artal and Tards, 2017) for the estimation of the camera trajectory and frame selection. Furthermore, we adopt the algorithm of Whelan et al (2015) for real-time fusion of depth maps in combination with color images.…”

“…In the second stage of our processing pipeline, we perform a multi-view dense image matching and depth estimation. In this, we use the efficient hierarchical plane-sweep multi-image matching with a subsequent surface-aware semi-global cost volume optimization (SGM sn ) proposed by Ruf et al (2019). This approach takes five images Ii of an input sequence, as well as the corresponding camera poses Pi, which we get from the camera tracking in the previous pipeline stage, and computes a depth map D, as well as a normal map N and a confidence map C for the middle one of the five input frames, i.e.…”

“…The scene space is sampled in such a way, that two consecutive planes introduce a maximum pixel displacement of 1 pixel along the epipolar line, leading to an increase in the sampling density the closer the plane is to the reference camera. In order to remedy a high memory consumption, Ruf et al (2019) employ for each pyramid level except the highest one, a dynamic cost volume and use the pixel-wise estimate of the previous level as a depth prior for a local sampling around the corresponding estimate.…”

“…A subsequent Semi-Global optimization scheme, as proposed by Hirschmueller (2008), is used in the second stage of the depth estimation in order to regularize the cost volume before extracting the depth map by selecting the pixel-wise winner-takes-it-all (WTA) solution. As stated by Ruf et al (2019), the initial Semi-Global Matching (SGM) algorithm (Hirschmueller, 2008) favors fronto-parallel surfaces. However, when considering oblique imagery, the scene mostly consists of slanted surfaces, which is why Ruf et al (2019) propose to use surface normals to shift the zero-cost transition within the path aggregation of the SGM optimization, in order to better account for slanted surfaces.…”

“…As stated by Ruf et al (2019), the initial Semi-Global Matching (SGM) algorithm (Hirschmueller, 2008) favors fronto-parallel surfaces. However, when considering oblique imagery, the scene mostly consists of slanted surfaces, which is why Ruf et al (2019) propose to use surface normals to shift the zero-cost transition within the path aggregation of the SGM optimization, in order to better account for slanted surfaces. We use this so-called SGM sn variant as part of the dense-image matching and depth estimation of our processing pipeline.…”