Learning to Fuse Proposals from Multiple Scanline Optimizations in Semi-Global Matching

Schönberger, Johannes L.; Sinha, Sudipta N.; Pollefeys, Marc

doi:10.1007/978-3-030-01261-8_45

Cited by 54 publications

(42 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has three SGA layers, two LGA layers and fifteen 3D convolutional layers for cost aggregation. (9,12), (7,14), (5,16), (3,18), (17,20), (15,22), (13,24), (11,26), ( from (3), 3×3 conv 1 /3H× 1 /3W×32 (7) 3×3 conv (no bn & relu) 1 /3H× 1 /3W×640 (8) split, reshape, normalize 4× 1 /3H× 1 /3W×5×32 (9)- (11) from (6), repeat (6)-(8) 4× 1 /3H× 1 /3W×5×32 (12) from (1), 3×3 conv H×W×16 (13) 3×3 conv (no bn & relu) H×W×75 (14) split, reshape, normalize H×W×75 (15)- (17) from (12), repeat (12)-(14) H×W×75 Cost Aggregation input 4D cost volume 1 /3H× 1 /3W×48×64 [1] 3×3×3, 3D conv 1 /3H× 1 /3W×48×32 [2] SGA layer: weight matrices from (5) 1 /3H× 1 /3W×48×32 [3] 3×3×3, 3D conv 1 /3H× 1 /3W×48×32 output 3×3×3, 3D to 2D conv, upsamping H×W×193 softmax, regression, loss weight: 0.2 H×W×1 [4] 3×3×3, 3D conv, stride 2 1 /6H× 1 /6W×48×48 [5] 3×3×3, 3D conv, stride 2 1 /12H× 1 /12W×48×64 [6] 3×3×3, 3D deconv, stride 2 1 /6H× 1 /6W×48×48 [7] 3×3×3, 3D conv 1 /6H× 1 /6W×48×48 [8] 3×3×3, 3D deconv, stride 2 1 /3H× 1 /3W×48×32 [9] 3×3×3, 3D conv 1 /3H× 1 /3W×48×32 [10] SGA layer: weight matrices from (8) 1 /3H× 1 /3W×48×32 output 3×3×3, 3D to 2D conv, upsamping H×W×193 softmax, regression, loss weight: 0.6 H×W×1 [11] 3×3×3, 3D conv...…”

Section: Resultsmentioning

confidence: 99%

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

Zhang

Prisacariu

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

694

512

View full text Add to dashboard Cite

In the stereo matching task, matching cost aggregation is crucial in both traditional methods and deep neural network models in order to accurately estimate disparities. We propose two novel neural net layers, aimed at capturing local and the whole-image cost dependencies respectively. The first is a semi-global aggregation layer which is a differentiable approximation of the semi-global matching, the second is the local guided aggregation layer which follows a traditional cost filtering strategy to refine thin structures.These two layers can be used to replace the widely used 3D convolutional layer which is computationally costly and memory-consuming as it has cubic computational/memory complexity. In the experiments, we show that nets with a two-layer guided aggregation block easily outperform the state-of-the-art GC-Net which has nineteen 3D convolutional layers. We also train a deep guided aggregation network (GA-Net) which gets better accuracies than state-of-the-art methods on both Scene Flow dataset and KITTI benchmarks. Code will be available at

show abstract

Section: Resultsmentioning

confidence: 99%

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

Zhang

Prisacariu

Yang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

694

512

View full text Add to dashboard Cite

show abstract

“…Batsos et al proposed CBMV [1] to combine evidence from multiple basic matching costs. Schonberger et al [19] proposed to classify scanline matching cost candidates with a random forest classifier. Seki et al proposed SGM-Nets [20] to provide learned penalties for SGM.…”

Section: Learning Based Methodsmentioning

confidence: 99%

“…GC-Net [6] and PSMNet [2] construct concatenation-based feature volume and incorporate a 3D CNN to aggregate contextual features. There are also works [1,19] trying to aggregate evidence from multiple hand-crafted matching cost proposals. However, the above methods have several drawbacks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Group-Wise Correlation Stereo Network

Guo

Yang²,

Yang³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

573

445

View full text Add to dashboard Cite

Stereo matching estimates the disparity between a rectified image pair, which is of great importance to depth sensing, autonomous driving, and other related tasks. Previous works built cost volumes with cross-correlation or concatenation of left and right features across all disparity levels, and then a 2D or 3D convolutional neural network is utilized to regress the disparity maps. In this paper, we propose to construct the cost volume by group-wise correlation. The left features and the right features are divided into groups along the channel dimension, and correlation maps are computed among each group to obtain multiple matching cost proposals, which are then packed into a cost volume. Group-wise correlation provides efficient representations for measuring feature similarities and will not lose too much information like full correlation. It also preserves better performance when reducing parameters compared with previous methods. The 3D stacked hourglass network proposed in previous works is improved to boost the performance and decrease the inference computational cost. Experiment results show that our method outperforms previous methods on Scene Flow, KITTI 2012, and KITTI 2015 datasets. The code is available at https: //github.com/xy-guo/GwcNet

show abstract

“…SGM has been applied in numerous fields, including building reconstruction, digital surface model generation, robot navigation, driver assistance and so forth [7][8][9]. However, the energy summation from all scanlines and the corresponding WTA strategy are empirical steps without a theoretical background, which is essentially inadequate when different scanlines propose inconsistent solutions [10]. Schönberger et al [10] proposed SGM-Forest, which trained a random forest to predict a scanline with the best disparity proposal.…”

Section: Introductionmentioning

confidence: 99%

Multi-Label Learning based Semi-Global Matching Forest

et al. 2020

View full text Add to dashboard Cite

Semi-Global Matching (SGM) approximates a 2D Markov Random Field (MRF) via multiple 1D scanline optimizations, which serves as a good trade-off between accuracy and efficiency in dense matching. Nevertheless, the performance is limited due to the simple summation of the aggregated costs from all 1D scanline optimizations for the final disparity estimation. SGM-Forest improves the performance of SGM by training a random forest to predict the best scanline according to each scanline’s disparity proposal. The disparity estimated by the best scanline acts as reference to adaptively adopt close proposals for further post-processing. However, in many cases more than one scanline is capable of providing a good prediction. Training the random forest with only one scanline labeled may limit or even confuse the learning procedure when other scanlines can offer similar contributions. In this paper, we propose a multi-label classification strategy to further improve SGM-Forest. Each training sample is allowed to be described by multiple labels (or zero label) if more than one (or none) scanline gives a proper prediction. We test the proposed method on stereo matching datasets, from Middlebury, ETH3D, EuroSDR image matching benchmark, and the 2019 IEEE GRSS data fusion contest. The result indicates that under the framework of SGM-Forest, the multi-label strategy outperforms the single-label scheme consistently.

show abstract

Learning to Fuse Proposals from Multiple Scanline Optimizations in Semi-Global Matching

Cited by 54 publications

References 43 publications

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

Group-Wise Correlation Stereo Network

Multi-Label Learning based Semi-Global Matching Forest

Contact Info

Product

Resources

About