Semantic segmentation-based stereo reconstruction with statistically improved long range accuracy

Miclea, Vlad-Cristian; Nedevschi, Sergiu

doi:10.1109/ivs.2017.7995967

Cited by 8 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We plan to add feedback from the downstream semantic network to the TPNET for the fine-tuning of the PC processing. Segmentation such as described by Miclea and Nedevschi [23] (vegetation, poorly textured pavement, thin wires, vertical poles) may be applied to TPNET and used to modify the interpretation of the raw 2D correlation of the tile clusters.…”

Section: Discussionmentioning

confidence: 99%

See far with TPNET: a Tile Processor and a CNN Symbiosis

Filippov,

Dzhimiev

2018

Preprint

View full text Add to dashboard Cite

Throughout the evolution of the neural networks more specialized cells were added to the set of basic building blocks. These cells aim to improve training convergence, increase the overall performance, and reduce the number of required labels, all while preserving the expressive power of the universal network. Inspired by the partitioning of the human visual perception system between the eyes and the cerebral cortex, we present TPNET, which offloads universal and application-specific CNN from the bulk processing of the high resolution pixel data and performs the translation-variant image correction while delegating all non-linear decision making to the network.In this work, we explore application of TPNET to 3D perception with a narrow-baseline (0.0001-0.0025) quad stereo camera and prove that a trained network provides a disparity prediction from the 2D phase correlation output by the Tile Processor (TP) that is twice as accurate as the prediction from a carefully hand-crafted algorithm. The TP in turn reduces the dimensions of the input features of the network and provides instrument-invariant and translationinvariant data, making real-time high resolution stereo 3D perception feasible and easing the requirement to have a complete end-to-end network.

show abstract

Section: Discussionmentioning

confidence: 99%

See far with TPNET: a Tile Processor and a CNN Symbiosis

Filippov,

Dzhimiev

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The method uses stochastic optimization based on genetic algorithms to optimally generate a bitstring for each particular segment class. As presented in our previous works [33], [34], a viable census mask usually covers a surface of maximum 15 × 15 pixels, giving enough information and allowing for maximum 32 pixels to be selected. Moreover, the computation time increases proportionally to each additional pixel, so larger census windows might lead to lower frame rates.…”

Section: B Cost Computationmentioning

confidence: 99%

Real-Time Semantic Segmentation-Based Stereo Reconstruction

Miclea

Nedevschi

2020

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a novel semantic segmentation-based stereo reconstruction method that can keep up with the accuracy of the state-of-the art approaches while running in real time. The solution follows the classic stereo pipeline, each step in the stereo workflow being enhanced by additional information from semantic segmentation. Therefore, we introduce several improvements to computation, aggregation, and optimization by adapting existing techniques to integrate additional surface information given by each semantic class. For the cost computation and optimization steps, we propose new genetic algorithms that can incrementally adjust the parameters for better solutions. Furthermore, we propose a new postprocessing edge-aware filtering technique relying on an improved convolutional neural network (CNN) architecture for disparity refinement. We obtain the competitive results at 30 frames/s, including segmentation.

show abstract

“…Other modifica-tions to SGM target its cost function and smoothness term to enhance the accuracy and coverage of the resulting depth estimates leaving the actual cost aggregation strategy untouched. This for instance includes the use of weighted similarity metrics (Miclea, Nedevschi, 2017), extended discontinuity penalties (Michael et al, 2013), their automatic adaptation to the image content (Karkalou et al, 2017), and combinations. Lately, machine learning techniques to predict SGM parameters have been discussed (Seki, Pollefeys, 2017).…”

Section: Introductionmentioning

confidence: 99%

Lock-Free Multithreaded Semi-Global Matching With an Arbitrary Number of Path Directions

Frommholz¹

2020

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. This paper describes an efficient implementation of the semi-global matching (SGM) algorithm on multi-core processors that allows a nearly arbitrary number of path directions for the cost aggregation stage. The scanlines for each orientation are discretized iteratively once, and the regular substructures of the obtained template are reused and shifted to concurrently sum up the path cost in at most two sweeps per direction over the disparity space image. Since path overlaps do not occur at any time, no expensive thread synchronization will be needed. To further reduce the runtime on high counts of path directions, pixel-wise disparity gating is applied, and both the cost function and disparity loop of SGM are optimized using current single instruction multiple data (SIMD) intrinsics for two major CPU architectures. Performance evaluation of the proposed implementation on synthetic ground truth reveals a reduced height error if the number of aggregation directions is significantly increased or when the paths start with an angular offset. Overall runtime shows a speedup that is nearly linear to the number of available processors.

show abstract

Semantic segmentation-based stereo reconstruction with statistically improved long range accuracy

Cited by 8 publications

References 30 publications

See far with TPNET: a Tile Processor and a CNN Symbiosis

See far with TPNET: a Tile Processor and a CNN Symbiosis

Real-Time Semantic Segmentation-Based Stereo Reconstruction

Lock-Free Multithreaded Semi-Global Matching With an Arbitrary Number of Path Directions

Contact Info

Product

Resources

About