Designing deep networks for surface normal estimation

Abstract

Input Image!Surface Normal (Output)! Input Image! Surface Normal (Output)! Figure 1: Given a single image, our algorithm estimates the surface normal at each pixel. Notice how our algorithm not only estimates the coarse structure also captures fine local details. For example, on the left, the normals of the couch arm and side table legs are estimated accurately (see zoomed version). On the right, the chair surface and legs and even the top of the shopping bags are captured correctly. Normal legend: blue → X; green → Y; red → Z. AbstractIn the past few years, convolutional neural nets (CNN) have shown incredible promise for learning visual representations. In this paper, we use CNNs for the task of predicting surface normals from a single image. But what is the right architecture we should use? We propose to build upon the decades of hard work in 3D scene understanding, to design new CNN architecture for the task of surface normal estimation. We show by incorporating several constraints (man-made, manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation. We also show that our network is quite robust and show state of the art results on other datasets as well without any fine-tuning.

Fast depth extraction from a single image

“…Eigen and Fergus 21 proposed to use a multiscale convolutional network to predict depth from a single image. Wang et al 22 presented a novel convolutional neural network (CNN) architecture for surface normal estimation. Liu et al 23,24 presented a deep convolutional neural field model based on fully convolutional networks and a novel superpixel pooling method, combining the strength of deep CNN and the continuous CRF into a unified CNNs framework.…”

Face Normals "In-the-Wild" Using Fully Convolutional Networks

“…In particular, motivated by the success of deep learning to various tasks including object detection, dense semantic segmentation, and normal estimation of scenes [23,2,33,62] etc., we propose to exploit the available large scale facial databases captured both in controlled, as well as in unconstrained conditions [8,48] to train a fully convolutional deep network that maps image pixels to normals. More precisely, to acquire accurate ground truth of facial normals we synthesise images of faces created with the use of recently released Large-Scale 3D Facial Models (LSFM) [8] which contains facial shapes of individuals with diverse ethnicities and characteristics.…”

Exploiting High Level Scene Cues in Stereo Reconstruction

“…At the local level, frequent relationships between small numbers of neighbouring oriented planes [8] and concave/convex edges [9] can provide a great deal of information about the scene. This idea has recently been extended to exploit a learned representation via convolutional neural networks [34]. This idea can be extended to interpreting different types of relationship between groups of primitives, such as "on-top-of", "supporting", "occluding" etc.…”