ActiveStereoNet: End-to-End Self-supervised Learning for Active Stereo Systems

Zhang, Yinda; Khamis, Sameh; Rhemann, Christoph; Valentin, Julien; Kowdle, Adarsh; Tankovich, Vladimir; Schoenberg, Michael; Izadi, Shahram; Funkhouser, Thomas; Fanello, Sean

doi:10.1007/978-3-030-01237-3_48

Cited by 100 publications

(99 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Leveraging this data, StereoNet [25] uses a hierarchical disparity prediction network with a deep visual feature backbone which is capable of running at 60 FPS on a consumer GPU. Its successor [56] extends the work with self-supervision to the domain of active sensing while maintaining the core efficiency. We build upon their work to leverage this computational advantage.…”

Section: Monocular Visionmentioning

confidence: 98%

“…To produce high quality depth maps, our architecture takes inspiration from two state-of-the-art pipelines for realtime disparity prediction, namely StereoNet [25] and Ac-tiveStereoNet [56] which estimate a subpixel precise lowresolution disparity map that is consecutively upsampled and refined with RGB-guidance from the reference image. Our depth estimation network consists of two Siamese towers with shared weights that extract deep image features at 1/8 of the stereo pair resolution following the architecture described in [56]. We construct a cost volume (CV) by concatenation of the displaced features along the epipolar lines of the rectified input images.…”

Section: Disparity Estimationmentioning

confidence: 99%

“…A pretrained StereoNet is utilized for which the weights are frozen before the cost volume computation step. The refinement network uses the same blocks as in a refinement scale of ActiveStereoNet [56]. We call this method cost volume refinement.…”

Section: Refocusing Architecturesmentioning

confidence: 99%

See 2 more Smart Citations

SteReFo: Efficient Image Refocusing with Stereo Vision

Busam

Hog

McDonagh

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Whether to attract viewer attention to a particular object, give the impression of depth or simply reproduce humanlike scene perception, shallow depth of field images are used extensively by professional and amateur photographers alike. To this end, high quality optical systems are used in DSLR cameras to focus on a specific depth plane while producing visually pleasing bokeh. We propose a physically motivated pipeline to mimic this effect from all-in-focus stereo images, typically retrieved by mobile cameras. It is capable to change the focal plane a posteriori at 76 FPS on KITTI [13] images to enable realtime applications. As our portmanteau suggests, SteReFo interrelates stereo-based depth estimation and refocusing efficiently. In contrast to other approaches, our pipeline is simultaneously fully differentiable, physically motivated, and agnostic to scene content. It also enables computational video focus tracking for moving objects in addition to refocusing of static images. We evaluate our approach on publicly available datasets [13,33,9] and quantify the quality of architectural changes.

show abstract

Section: Monocular Visionmentioning

confidence: 98%

Section: Disparity Estimationmentioning

confidence: 99%

See 1 more Smart Citation

SteReFo: Efficient Image Refocusing with Stereo Vision

Busam

Hog

McDonagh

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

show abstract

“…The proposed refinement network described in Table. 7 is inspired by the refinement procedures proposed in CRL [17], iResNet [12], StereoNet [8], and ActiveStere-oNet [31]. We adopted the basic architecture for refinement as described in StereoNet [8] with dilated residual blocks [28] to increase the receptive field of filtering without compromising resolution.…”

Section: D Dilation In Cost Filteringmentioning

confidence: 99%

StereoDRNet: Dilated Residual StereoNet

Chabra

Straub

Sweeney

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Figure 1: StereoDRNet enables estimation of high quality depth maps that opens the door to high quality reconstruction by passive stereo video. In this figure we compare the output from dense reconstruction [15] built form depth maps generated by StereoDRNet, PSMNet [2] and a structured light system [25] (termed Ground Truth). We report and visualize point-toplane distance RMS error on the reconstructed meshes with respect to the ground truth demonstrating the improvement in reconstruction over the state-of-the-art. AbstractWe propose a system that uses a convolution neural network (CNN) to estimate depth from a stereo pair followed by volumetric fusion of the predicted depth maps to produce a 3D reconstruction of a scene. Our proposed depth refinement architecture, predicts view-consistent disparity and occlusion maps that helps the fusion system to produce geometrically consistent reconstructions. We utilize 3D dilated convolutions in our proposed cost filtering network that yields better filtering while almost halving the computational cost in comparison to state of the art cost filtering architectures. For feature extraction we use the Vortex Pooling architecture [26]. The proposed method achieves state of the art results in KITTI 2012, KITTI 2015 and ETH 3D stereo benchmarks. Finally, we demonstrate that our system is able to produce high fidelity 3D scene reconstructions that outperforms the state of the art stereo system.

show abstract

“…Inspired by recent progress in self-supervised depth [5], [41], flow [12], [20] and representation learning [3], [24], we propose a novel approach for self-supervised image deblurring which only relies on real-world blurry image sequences for training. Self-supervised learning improves the network's generalization performance, by enabling the network to adapt to scenarios where ground truth sharp images are not available.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Linear Motion Deblurring

Liu

Janai

Pollefeys

et al. 2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Motion blurry images challenge many computer vision algorithms, e.g., feature detection, motion estimation, or object recognition. Deep convolutional neural networks are stateof-the-art for image deblurring. However, obtaining training data with corresponding sharp and blurry image pairs can be difficult. In this paper, we present a differentiable reblur model for selfsupervised motion deblurring, which enables the network to learn from real-world blurry image sequences without relying on sharp images for supervision. Our key insight is that motion cues obtained from consecutive images yield sufficient information to inform the deblurring task. We therefore formulate deblurring as an inverse rendering problem, taking into account the physical image formation process: we first predict two deblurred images from which we estimate the corresponding optical flow. Using these predictions, we re-render the blurred images and minimize the difference with respect to the original blurry inputs. We use both synthetic and real dataset for experimental evaluations. Our experiments demonstrate that self-supervised single image deblurring is really feasible and leads to visually compelling results. Both the code and datasets are available at https://github.com/ethliup/SelfDeblur.

show abstract

ActiveStereoNet: End-to-End Self-supervised Learning for Active Stereo Systems

Cited by 100 publications

References 55 publications

SteReFo: Efficient Image Refocusing with Stereo Vision

SteReFo: Efficient Image Refocusing with Stereo Vision

StereoDRNet: Dilated Residual StereoNet

Self-Supervised Linear Motion Deblurring

Contact Info

Product

Resources

About