PGNet: Panoptic parsing guided deep stereo matching

Chen, Shuya; Xiang, Zhiyu; Qiao, Chengyu; Chen, Yiman; Bai, Tingming

doi:10.1016/j.neucom.2021.08.041

Cited by 16 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A semantic segmentation branch was proposed in the SegStereo 9 to incorporate additional semantic information to stereo matching tasks. PG-Net 13 proposed a panoptic parsing guided deep network to solve the stereo matching tasks. PDSNet 34 introduces a bottleneck matching module that enhances the ability to utilize global feature information.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An application of stereo matching algorithm based on transfer learning on robots in multiple scenarios

Wang

2023

Preprint

View full text Add to dashboard Cite

Robot vision technology based on binocular vision has significant potential for development in various fields, including 3D scenes and reconstruction, target detection, autonomous driving, and other fields. To date, current binocular vision methods used in robotics engineering suffer from high costs, complex algorithms, and low reliability of the generated disparity map in multiple scenarios. Robots require a cost-effective algorithm with cross-domain generalization capabilities for multiple scenarios. To address these issues, a cross-domain stereo matching algorithm for binocular vision based on transfer learning was proposed in this paper, named Cross-Domain Adaptation and Transfer Learning Network (Ct-Net), which has shown valuable results in multiple robot scenes. First, this paper introduces a General Feature Extractor (GFE) to extract rich general feature information for domain adaptive stereo matching tasks. Then, a feature adapter is used to adapt the general features to the stereo matching network. Furthermore, a Domain Adaptive Cost Optimization Module (DACOM) was designed to optimize the matching cost. A disparity score prediction module was also embedded to adaptively adjust the search range of disparity and optimize the cost distribution. The overall framework was trained using a phased strategy, and ablation experiments were conducted to verify the effectiveness of the training strategy. On KITTI 2015 benchmark, compared with the prototype PSMNet, the 3PE − fg of Ct-Net in all regions and non-occluded regions decreased by 19.3% and 21.1% respectively. On the Middlebury dataset, the 2PE of Ct-Net achieved comparable results on all samples. The quantitative and qualitative results obtained from Middlebury, Apollo, and other datasets demonstrate that Ct-Net significantly improves the cross-domain performance of stereo matching. Stereo matching experiments in real-world scenarios have shown that it can effectively address visual tasks in multiple robot scenes.

show abstract

Section: Related Workmentioning

confidence: 99%

“…A semantic segmentation branch is proposed in work 12 to incorporate additional semantic information into stereo matching tasks. PGNet 13 proposed a panoptic parsing guided deep network to solve the stereo matching task. A cascading fusion cost volume is proposed to optimize the cost distribution 14 .…”

Section: Introductionmentioning

confidence: 99%

An application of stereo matching algorithm based on transfer learning on robots in multiple scenarios

Wang

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The flood filling algorithm [16][17] is an algorithm that simulates the fluid diffusion process to fill the connected region with a specific color, which is widely used in image processing to determine the connected region in two-dimensional space. Starting from a selected pixel point F(x,y), Based on the difference between the gray value of the pixel point and the surrounding pixel points to determine whether the surrounding target pixel points belong to the joint area, to satisfy the judgment, the point is filled, and the target point is used as a new seed point for the next round of filling process, until the joint area is completely filled.…”

Section: Flood Filling Algorithmmentioning

confidence: 99%

“…Although it better neutralizes the advantages and disadvantages of local and global matching, it is weak in dealing with occluded regions as well as parallax discontinuity regions, and large deviations occur in places of occluded regions. In recent years, due to the rapid development of deep learning, Cao Y et al proposed AMDCNet, PGNet,PSNet and other stereo matching methods [16][17][18], although the methods can solve the above problems well and the matching accuracy is also high, but this type of methods need to rely on a large number of training datasets, stereo matching datasets are difficult and costly to collect, and the application scenarios are also too limited. Therefore this class of methods cannot be applied well to practical scenarios.…”

Section: Introductionmentioning

confidence: 99%

Binocular Stereo Matching Based on SGM and Flood Filling Algorithm

Song

Shu

et al. 2022

Preprint

View full text Add to dashboard Cite

The SGM algorithm was weak in matching the occluded area and parallax discontinuity area in stereo matching, and it was easy to produce more voids. To address this question, the paper proposes a parallax optimizati-on algorithm based on SGM algorithm combined with flood filling and median smoothing. The algorithm fills the missing parallax values based on the parallax values at the edges of the voids and achieves the repair of the missing parallax in the parallax map without raising a large computational effort. The proposed algorithm has also been qua-ntitatively analyzed on the Middlebury stereo matching dataset using three evaluation metrics: root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity (MSSIM). The experimental results show that the improvement and optimization of the SGM algorithm can effectively fill the voids in the parallax map and reduce the noise of the image, and the matching accuracy of parallax maps is improved by an average of 9%.

show abstract

Section: Introductionmentioning

confidence: 99%

An application of stereo matching algorithm based on transfer learning on robots in multiple scenes

Bi,

Li,

Tong

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Robot vision technology based on binocular vision holds tremendous potential for development in various fields, including 3D scene reconstruction, target detection, and autonomous driving. However, current binocular vision methods used in robotics engineering have limitations such as high costs, complex algorithms, and low reliability of the generated disparity map in different scenes. To overcome these challenges, a cross-domain stereo matching algorithm for binocular vision based on transfer learning was proposed in this paper, named Cross-Domain Adaptation and Transfer Learning Network (Ct-Net), which has shown valuable results in multiple robot scenes. First, this paper introduces a General Feature Extractor to extract rich general feature information for domain adaptive stereo matching tasks. Then, a feature adapter is used to adapt the general features to the stereo matching network. Furthermore, a Domain Adaptive Cost Optimization Module is designed to optimize the matching cost. A disparity score prediction module was also embedded to adaptively adjust the search range of disparity and optimize the cost distribution. The overall framework was trained using a phased strategy, and ablation experiments were conducted to verify the effectiveness of the training strategy. Compared with the prototype PSMNet, on KITTI 2015 benchmark, the 3PE-fg of Ct-Net in all regions and non-occluded regions decreased by 19.3 and 21.1% respectively, meanwhile, on the Middlebury dataset, the proposed algorithm improves the sample error rate at least 28.4%, which is the Staircase sample. The quantitative and qualitative results obtained from Middlebury, Apollo, and other datasets demonstrate that Ct-Net significantly improves the cross-domain performance of stereo matching. Stereo matching experiments in real-world scenes have shown that it can effectively address visual tasks in multiple scenes.

show abstract

PGNet: Panoptic parsing guided deep stereo matching

Cited by 16 publications

References 13 publications

An application of stereo matching algorithm based on transfer learning on robots in multiple scenarios

An application of stereo matching algorithm based on transfer learning on robots in multiple scenarios

Binocular Stereo Matching Based on SGM and Flood Filling Algorithm

An application of stereo matching algorithm based on transfer learning on robots in multiple scenes

Contact Info

Product

Resources

About