GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation

Qi, Xiaojuan; Liao, Renjie; Liu, Zhengzhe; Urtasun, Raquel; Jia, Jiaya

doi:10.1109/cvpr.2018.00037

Cited by 363 publications

(298 citation statements)

References 20 publications

Supporting

Mentioning

288

Contrasting

Unclassified

Order By: Relevance

“…The loss function is the mean angular difference between the ground-truth and the regressed normal. [30] (from top to bottom). Right: sectional view of these results.…”

Section: Normal Networkmentioning

confidence: 99%

“…The above shape refinement is iterated for 5 times in our network to simulate the iterative solution of the original energy equation in [24]. Figure 3 compares our method with the 'Kernel Regression' layer [30] on a toy example, which is also designed to fuse the surface normal and depth. Figure 4 shows a comparison with the work [24] on real data and our method also produces more convincing result.…”

Section: Depth Refinementmentioning

confidence: 99%

See 1 more Smart Citation

A Neural Network for Detailed Human Depth Estimation From a Single Image

Tang

Tan

Cheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

This paper presents a neural network to estimate a detailed depth map of the foreground human in a single RGB image. The result captures geometry details such as cloth wrinkles, which are important in visualization applications. To achieve this goal, we separate the depth map into a smooth base shape and a residual detail shape and design a network with two branches to regress them respectively. We design a training strategy to ensure both base and detail shapes can be faithfully learned by the corresponding network branches. Furthermore, we introduce a novel network layer to fuse a rough depth map and surface normals to further improve the final result. Quantitative comparison with fused 'ground truth' captured by real depth cameras and qualitative examples on unconstrained Internet images demonstrate the strength of the proposed method. Our code will be released at Link

show abstract

“…The loss function is the mean angular difference between the ground-truth and the regressed normal. [30] (from top to bottom). Right: sectional view of these results.…”

Section: Normal Networkmentioning

confidence: 99%

Section: Depth Refinementmentioning

confidence: 99%

A Neural Network for Detailed Human Depth Estimation From a Single Image

Tang

Tan

Cheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…For example, a continuous conditional random field (CRF) [28] is used for depth prediction, which takes pair-wise information into account. Other high-order geometric relations [9,31] are also exploited, such as designing a gravity constraint for local regions [9] or incorporating the depth-to-surface-normal mutual transformation inside the optimization pipeline [31]. Note that, for the above methods, almost all the geometric constraints are 'local' in the sense that they are extracted from a small neighborhood in either 2D or 3D.…”

Section: Introductionmentioning

confidence: 99%

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

Yin

Liu

Shen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

429

333

View full text Add to dashboard Cite

Monocular depth prediction plays a crucial role in understanding 3D scene geometry. Although recent methods have achieved impressive progress in evaluation metrics such as the pixel-wise relative error, most methods neglect the geometric constraints in the 3D space. In this work, we show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces one simple type of geometric constraints, namely, virtual normal directions determined by randomly sampled three points in the reconstructed 3D space, we can considerably improve the depth prediction accuracy. Significantly, the byproduct of this predicted depth being sufficiently accurate is that we are now able to recover good 3D structures of the scene such as the point cloud and surface normal directly from the depth, eliminating the necessity of training new sub-models as was previously done. Experiments on two benchmarks: NYU Depth-V2 and KITTI demonstrate the effectiveness of our method and state-of-the-art performance.

show abstract

“…Each scene contains aligned RGB and depth images, acquired from a Microsoft Kinect sensor. Following previous works on single-image depth estimation [Chen et al 2016;Qi et al 2018;Zoran et al 2015], we use the standard training-testing split and evaluate our method on the 654 imagedepth pairs from the testing set.…”

Section: Depth Prediction Qualitymentioning

confidence: 99%

3D Ken Burns effect from a single image

et al. 2019

View full text Add to dashboard Cite

from to Fig. 1. 3D Ken Burns effect from a single image. Given a single input image and optional user annotations in form of two cropping windows, our framework animates the input image while adding parallax to synthesize a 3D Ken Burns effect. Our method works well for a wide variety of content, including portrait (top) and landscape (bottom) photos. Please refer to our supplementary video demo for more examples. Please note that this figure, as well as many other figures in this paper, contain video clips. Should these videos not already be playing then please consider viewing this paper using Adobe Reader.The Ken Burns effect allows animating still images with a virtual camera scan and zoom. Adding parallax, which results in the 3D Ken Burns effect, enables significantly more compelling results. Creating such effects manually is timeconsuming and demands sophisticated editing skills. Existing automatic methods, however, require multiple input images from varying viewpoints. In this paper, we introduce a framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework first leverages a depth prediction pipeline, which estimates scene depth that is suitable for view synthesis tasks. To address the limitations of existing depth estimation methods such as geometric distortions, semantic distortions, and inaccurate depth boundaries, we develop a semantic-aware neural network for depth prediction, couple its estimate with a segmentation-based depth adjustment process, and employ a refinement neural network that facilitates accurate depth predictions at object boundaries. According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions. To address disocclusions while maintaining geometrically and temporally coherent synthesis results, we utilize contextaware color-and depth-inpainting to fill in the missing information in the extreme views of the camera path, thus extending the scene geometry of the point cloud. Experiments with a wide variety of image content show that our method enables realistic synthesis results. Our study demonstrates that our system allows users to achieve better results while requiring little effort compared to existing solutions for the 3D Ken Burns effect creation.

show abstract

GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation

Cited by 363 publications

References 20 publications

A Neural Network for Detailed Human Depth Estimation From a Single Image

A Neural Network for Detailed Human Depth Estimation From a Single Image

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

3D Ken Burns effect from a single image

Contact Info

Product

Resources

About