Unsupervised Cross-Spectral Stereo Matching by Learning to Synthesize

Abstract

Unsupervised cross-spectral stereo matching aims at recovering disparity given cross-spectral image pairs without any depth or disparity supervision. The estimated depth provides additional information complementary to original images, which can be helpful for other vision tasks such as tracking, recognition and detection. However, there are large appearance variations between images from different spectral bands, which is a challenge for cross-spectral stereo matching. Existing deep unsupervised stereo matching methods are sensitive to the appearance variations and do not perform well on cross-spectral data. We propose a novel unsupervised crossspectral stereo matching framework based on image-to-image translation. First, a style adaptation network transforms images across different spectral bands by cycle consistency and adversarial learning, during which appearance variations are minimized. Then, a stereo matching network is trained with image pairs from the same spectra using view reconstruction loss. At last, the estimated disparity is utilized to supervise the spectral translation network in an end-to-end way. Moreover, a novel style adaptation network F-cycleGAN is proposed to improve the robustness of spectral translation. Our method can tackle appearance variations and enhance the robustness of unsupervised cross-spectral stereo matching. Experimental results show that our method achieves good performance without using depth supervision or explicit semantic information.

Degradation-agnostic Correspondence from Resolution-asymmetric Stereo

“…To estimate correspondence from stereo images with radiometric variation, different robust matching costs are proposed, such as mutual information measure [11] and adaptive normalized cross-correlation [12]. For cross-modal stereo [40,45,49], images from two different modalities are normalized to a single one to make up the photometric inconsistency, e.g., through deep transformation networks [21,49]. Recently, stereo matching with visual imbalance (monocular blur and noise) is addressed by integrating a view synthesis network and a stereo reconstruction network, which requires the ground-truth disparity, the high-quality version of the degraded view, and the explicit degradation form for supervision [25].…”

There and Back Again: Self-supervised Multispectral Correspondence Estimation

“…Generative approaches for RGB and NIR use a cycleGAN to match across generated stereo pairs for both spectra [42], [43]. Performance of these image-to-image translation approaches are subject to spectral similarity.…”

“…Both neural network approaches, from Zhi et al [31] and Liang et al [42] leverage image-to-image translation. This is possible because the majority of materials in the NIR spectrum closely resemble their grayscale counterparts in the visible image.…”

Matching-space Stereo Networks for Cross-domain Generalization

“…Most deep stereo models are particularly data dependent and their performance drops considerably when dealing with unseen domains different from those observed during training [37,38]. To tackle the domain shift problem, two main strategies are involved: image synthesis [6,14,18], and un-/selfsupervised adaptation [3,10,25,37,38,39,40,48,49]. In contrast, our method aims at being transferred without adaptation to different domains, being this possibility more appealing for practical applications.…”