Deep Material-Aware Cross-Spectral Stereo Matching

Zhi, Tiancheng; Pires, Bernardo R.; Hebert, Martial; Narasimhan, Srinivasa G.

doi:10.1109/cvpr.2018.00205

Cited by 47 publications

(50 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(aux) represents using the auxiliary loss during training. The STN(F) + SMN(aux)(ori) means the original image pairs instead of (Zhi et al 2018), where the DMC(w. seg.)…”

Section: Benchmark Resultsmentioning

confidence: 99%

“…Then, the absolute difference between the warped image and the left image, also called reconstruction error or photometric loss, is minimized to supervise disparity predictions. Godard et al (Godard, Mac Aodha, and Brostow 2017) (Zhi et al 2018). The proposed method performs well on the challenging materials like clothes (row 1,2), vegetation (row 2), lights (row 3).…”

Section: Related Work Unsupervised Depth Estimationmentioning

confidence: 91%

“…For deep learning methods, Aguilera et al (Aguilera et al 2016) learned a similarity measurement of cross-spectral image patches, which is a potential way to learn matching cost for multi-spectrum images. Zhi et al (Zhi et al 2018) utilized deep segmentation maps to improve robustness of cross-spectral stereo matching, while the method required extra semantic annotations and manually designed losses for different materials, which made it hard to apply to other scenes.…”

Section: Cross-spectral Stereo Matchingmentioning

confidence: 99%

“…reflection or transparent surfaces) (Chiu, Blanke, and Fritz 2011). However, the cross-spectral stereo matching is still a challenging task especially without depth supervision (Zhi et al 2018), because there are great illumination differences be-F G Stereo Matching Network " from spectrum A # from spectrum B " and % ( ( # )) ) ( ( " )) and # Disparity # Disparity "…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Unsupervised Cross-Spectral Stereo Matching by Learning to Synthesize

Liang

Guo

et al. 2019

AAAI

View full text Add to dashboard Cite

Unsupervised cross-spectral stereo matching aims at recovering disparity given cross-spectral image pairs without any depth or disparity supervision. The estimated depth provides additional information complementary to original images, which can be helpful for other vision tasks such as tracking, recognition and detection. However, there are large appearance variations between images from different spectral bands, which is a challenge for cross-spectral stereo matching. Existing deep unsupervised stereo matching methods are sensitive to the appearance variations and do not perform well on cross-spectral data. We propose a novel unsupervised crossspectral stereo matching framework based on image-to-image translation. First, a style adaptation network transforms images across different spectral bands by cycle consistency and adversarial learning, during which appearance variations are minimized. Then, a stereo matching network is trained with image pairs from the same spectra using view reconstruction loss. At last, the estimated disparity is utilized to supervise the spectral translation network in an end-to-end way. Moreover, a novel style adaptation network F-cycleGAN is proposed to improve the robustness of spectral translation. Our method can tackle appearance variations and enhance the robustness of unsupervised cross-spectral stereo matching. Experimental results show that our method achieves good performance without using depth supervision or explicit semantic information.

show abstract

“…(aux) represents using the auxiliary loss during training. The STN(F) + SMN(aux)(ori) means the original image pairs instead of (Zhi et al 2018), where the DMC(w. seg.)…”

Section: Benchmark Resultsmentioning

confidence: 99%

Section: Related Work Unsupervised Depth Estimationmentioning

confidence: 91%

Section: Cross-spectral Stereo Matchingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Cross-Spectral Stereo Matching by Learning to Synthesize

Liang

Guo

et al. 2019

AAAI

View full text Add to dashboard Cite

show abstract

“…Recently, also cross-spectral stereo camera approaches came up that combine, e.g., a color and an infrared camera for measuring multiple spectral components [42]. As image content is recorded at different spatial positions, the aim is to register heterogeneous content.…”

Section: Related Multi-spectral Imaging Techniquesmentioning

confidence: 99%

Camera Array for Multi-Spectral Imaging

Genser

Seiler

Kaup

2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Recently, many new applications arose for multispectral and hyper-spectral imaging. Besides modern biometric systems for identity verification, also agricultural and medical applications came up, which measure the health condition of plants and humans. Despite the growing demand, the acquisition of multi-spectral data is up to the present complicated. Often, expensive, inflexible, or low resolution acquisition setups are only obtainable for specific professional applications. To overcome these limitations, a novel camera array for multi-spectral imaging is presented in this article for generating consistent multispectral videos. As differing spectral images are acquired at various viewpoints, a geometrically constrained multi-camera sensor layout is introduced, which enables the formulation of novel registration and reconstruction algorithms to globally set up robust models. On average, the novel acquisition approach achieves a gain of 2.5 dB PSNR compared to recently published multi-spectral filter array imaging systems. At the same time, the proposed acquisition system ensures not only a superior spatial, but also a high spectral, and temporal resolution, while filters are flexibly exchangeable by the user depending on the application. Moreover, depth information is generated, so that 3D imaging applications, e.g., for augmented or virtual reality, become possible. The proposed camera array for multi-spectral imaging can be set up using off-the-shelf hardware, which allows for a compact design and employment in, e.g., mobile devices or drones, while being cost-effective.

show abstract

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Tosi

Poggi

Benincasa

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Confidence measures for stereo gained popularity in recent years due to their improved capability to detect outliers and the increasing number of applications exploiting these cues. In this field, convolutional neural networks achieved top-performance compared to other known techniques in the literature by processing local information to tell disparity assignments from outliers. Despite this outstanding achievements, all approaches rely on clues extracted with small receptive fields thus ignoring most of the overall image content. Therefore, in this paper, we propose to exploit nearby and farther clues available from image and disparity domains to obtain a more accurate confidence estimation. While local information is very effective for detecting high frequency patterns, it lacks insights from farther regions in the scene. On the other hand, enlarging the receptive field allows to include clues from farther regions but produces smoother uncertainty estimation, not particularly accurate when dealing with high frequency patterns. For these reasons, we propose in this paper a multi-stage cascaded network to combine the best of the two worlds. Extensive experiments on three datasets using three popular stereo algorithms prove that the proposed framework outperforms state-of-the-art confidence estimation techniques.

show abstract

Deep Material-Aware Cross-Spectral Stereo Matching

Cited by 47 publications

References 24 publications

Unsupervised Cross-Spectral Stereo Matching by Learning to Synthesize

Unsupervised Cross-Spectral Stereo Matching by Learning to Synthesize

Camera Array for Multi-Spectral Imaging

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Contact Info

Product

Resources

About