Deep Depth from Defocus: How Can Defocus Blur Improve 3D Estimation Using Dense Neural Networks?

Abstract

Depth estimation is of critical interest for scene understanding and accurate 3D reconstruction. Most recent approaches in depth estimation with deep learning exploit geometrical structures of standard sharp images to predict corresponding depth maps. However, cameras can also produce images with defocus blur depending on the depth of the objects and camera settings. Hence, these features may represent an important hint for learning to predict depth. In this paper, we propose a full system for single-image depth prediction in the wild using depth-fromdefocus and neural networks. We carry out thorough experiments to test deep convolutional networks on real and simulated defocused images using a realistic model of blur variation with respect to depth. We also investigate the influence of blur on depth prediction observing model uncertainty with a Bayesian neural network approach. From these studies, we show that out-of-focus blur greatly improves the depth-prediction network performances. Furthermore, we transfer the ability learned on a synthetic, indoor dataset to real, indoor and outdoor images. For this purpose, we present a new dataset containing real all-focus and defocused images from a Digital Single-Lens Reflex (DSLR) camera, paired with ground truth depth maps obtained with an active 3D sensor for indoor scenes. The proposed approach is successfully validated on both this new dataset and standard ones as NYUv2 or Depth-in-the-Wild. Code and new datasets are available at https:// github.com/ marcelampc/ d3net depth estimation.

Single Image Depth Estimation Trained via Depth From Defocus Cues

“…From the lightfield images, we follow the procedure of [31] to generate the all-in-focus and shallow DoF images, and split the dataset into 3143 and 300 images for train and test. DSLR dataset [3] This dataset contains 110 images and ground truth depth from indoor scenes, with 81 images for training and 29 images for testing, and 34 images from outdoor scenes without ground truth depth. Each scene is ac- Make3D [27,28] The Make3D benchmark contains 534 RGB-depth pairs, split into 400 pairs for training and 134 for testing.…”

“…where D o is the distance between an object to the lens plane, and A = F/N where N is what is known as the f-number of the camera. While CoC is usually measured in millimeters (C mm ), we transform its size to pixels by considering a camera pixel-size of p = 5.6µm as in [3], and a camera output scale s, which is the ratio between sensor size and output image size. The final CoC size in pixels C is computed as follows:…”

“…For rendered DoF supervision, we consider four datasets [8,27,23,3] with ground truth depth, where we render focused images with different focus distances. We denote by F1, F2, F6, F10 the four training setups, which differ by the number of rendered focused images used in training.…”

“…Quantitative results on the outdoor DSLR[3] test set, reported as mean value of PSNR and SSIM of the reconstructed focused image.…”

Deep Optics for Monocular Depth Estimation and 3D Object Detection

“…However, performance is highly dependent on the training dataset. To address this issue, several recent approaches have incorporated physical camera parameters into their image formation model, including focal length [14] and defocus blur [1], to implicitly encode 3D information into a 2D image. We build on these previous insights and perform a significantly more extensive study that evaluates several types of fixed lenses as well as fully optimizable camera lenses for monocular depth estimation and 3D object detection tasks.…”