Single-Shot Analysis of Refractive Shape Using Convolutional Neural Networks

Abstract

The appearance of a transparent object is determined by a combination of refraction and reflection, as governed by a complex function of its shape as well as the surrounding environment. Prior works on 3D reconstruction have largely ignored transparent objects due to this challenge, yet they occur frequently in real-world scenes. This paper presents an approach to estimate depths and normals for transparent objects using a single image acquired under a distant but otherwise arbitrary environment map. In particular, we use a deep convolutional neural network (CNN) for this task. Unlike opaque objects, it is challenging to acquire ground truth training data for refractive objects, thus, we propose to use a large-scale synthetic dataset. To accurately capture the image formation process, we use a physically-based renderer. We demonstrate that a CNN trained on our dataset learns to reconstruct shape and estimate segmentation boundaries for transparent objects using a single image, while also achieving generalization to real images at test time. In experiments, we extensively study the properties of our dataset and compare to baselines demonstrating its utility.

A Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction

“…In addition to optimization-based methods, deep learning techniques can also be incorporated to resolve depth-normal ambiguity. Stets et al [2019] and Sajjan et al [2020] proposed the use of a pre-trained encoder-decoder network to estimate mask, depth, and normals from a single image. suggested performing optimization in feature space to obtain surface normals.…”

Differentiable refraction-tracing for mesh reconstruction of transparent objects

“…More recently, several researchers tackle the task of transparent object reconstruction by incorporating deep learning techniques. Stets et al [2019] and Sajjan et al [2020] propose to use encoderdecoder architectures for estimating the segmentation mask, depth map and surface normals from a single input image of a transparent object. Li et al [2020] present a different approach, where a rendering layer is embedded in the network to account for complex light transport behaviors.…”

ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation

“…specular highlights and caustics. Datasets that do contain transparent objects have been used to study refractive flow estimation Chen et al [8], semantic segmentation [47], or relative depth [46]. These datasets are generated in a simplified setting (e.g.…”