Inferring Point Clouds from Single Monocular Images by Depth Intermediation

Zeng, Wei; Karaoglu, Sezer; Gevers, Theo

doi:10.48550/arxiv.1812.01402

Cited by 3 publications

(7 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is well suited for objects with intriguing parts and fine details. As such, an increasing number of papers, at least one in 2017 [68], more than 12 in 2018 [21], [21], [22], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], and a few others in 2019 [77], explored their usage for deep learningbased reconstruction. This section discusses the stateof-the-art point-based representations and their corresponding network architectures.…”

Section: Point-based Techniquesmentioning

confidence: 99%

“…While early methods train separately the different modules, recent works proposed end-to-end solutions [7], [10], [37], [49], [76], [87], [88]. For instance, Wu et al [7] and later Sun et al [10] used two blocks.…”

Section: Intermediatingmentioning

confidence: 99%

“…Other techniques convert the intermediate depth map into (1) a 3D occupancy grid [42] or a truncated signed distance function volume [37], which is then processed using a 3D encoder-decoder network for completion and refinement, or (2) a partial point cloud, which is further processed using a point-cloud completion module [76]. Zhang et al [88] convert the inferred depth map into a spherical map and unpaint it, to fill in holes, using another encoder-decoder.…”

Section: Intermediatingmentioning

confidence: 99%

See 2 more Smart Citations

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Han

Laga

Bennamoun

2021

IEEE Trans. Pattern Anal. Mach. Intell.

320

150

View full text Add to dashboard Cite

3D reconstruction is a longstanding ill-posed problem, which has been explored for decades by the computer vision, computer graphics, and machine learning communities. Since 2015, image-based 3D reconstruction using convolutional neural networks (CNN) has attracted increasing interest and demonstrated an impressive performance. Given this new era of rapid evolution, this article provides a comprehensive survey of the recent developments in this field. We focus on the works which use deep learning techniques to estimate the 3D shape of generic objects either from a single or multiple RGB images. We organize the literature based on the shape representations, the network architectures, and the training mechanisms they use. While this survey is intended for methods which reconstruct generic objects, we also review some of the recent works which focus on specific object classes such as human body shapes and faces. We provide an analysis and comparison of the performance of some key papers, summarize some of the open problems in this field, and discuss promising directions for future research.

show abstract

Section: Point-based Techniquesmentioning

confidence: 99%

Section: Intermediatingmentioning

confidence: 99%

Section: Intermediatingmentioning

confidence: 99%

See 1 more Smart Citation

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Han

Laga

Bennamoun

2021

IEEE Trans. Pattern Anal. Mach. Intell.

320

150

View full text Add to dashboard Cite

show abstract

“…Such point-based representations are well suited for objects with intriguing parts and fine details. As such, an increasing number of papers, e.g., [9,10,11,12,13,14,15,39,16,17,18,20,19], explored their usage for deep learning-based reconstruction. To reconstruct point clouds from an input image, these methods also use an encoder, similar to volumetric representations.…”

Section: Related Workmentioning

confidence: 99%

“…In these works, 3D reconstruction is formulated as an inference problem taking advantage of the availability of collections of images annotated with their corresponding 3D models [3,4]. Existing deep learning methods for 3D reconstruction represent 3D models as volumes [5,6,7,8], point clouds [9,10,11,12,13,14,15,16,17,18,19,20], or triangulated meshes [21,22,23,24]. Volumetric representations are suitable for convolutional operations, which operate on regular grids.…”

Section: Introductionmentioning

confidence: 99%

Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images

Charrada,

Tabia,

Chetouani

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a novel deep reinforcement learning-based approach for 3D object reconstruction from monocular images. Prior works that use mesh representations are template based. Thus, they are limited to the reconstruction of objects that have the same topology as the template. Methods that use volumetric grids as intermediate representations are computationally expensive, which limits their application in real-time scenarios. In this paper, we propose a novel end-to-end method that reconstructs 3D objects of arbitrary topology from a monocular image. It is composed of of (1) a Vertex Generation Network (VGN), which predicts the initial 3D locations of the object's vertices from an input RGB image, (2) a differentiable triangulation layer, which learns in a non-supervised manner, using a novel reinforcement learning algorithm, the best triangulation of the object's vertices, and finally, (3) a hierarchical mesh refinement network that uses graph convolutions to refine the initial mesh. Our key contribution is the learnable triangulation process, which recovers in an unsupervised manner the topology of the input shape. Our experiments on ShapeNet and Pix3D benchmarks show that the proposed method outperforms the state-of-the-art in terms of visual quality, reconstruction accuracy, and computational time.

show abstract

Can Language Understand Depth?

Zhang¹,

Zeng²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Besides image classification, Contrastive Language-Image Pretraining (CLIP) has accomplished extraordinary success for a wide range of vision tasks, including object-level and 3D space understanding. However, it's still challenging to transfer semantic knowledge learned from CLIP into more intricate tasks of quantified targets, such as depth estimation with geometric information. In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of input image could respond to a certain semantic distance token and then be projected to a quantified depth bin for coarse estimation. Without any training, our DepthCLIP surpasses existing unsupervised methods and even approaches the early fully-supervised networks. To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation. We hope our work could cast a light on the future research. The code is available at https://github.com/Adonis-galaxy/DepthCLIP. CCS CONCEPTS• Computing methodologies → Neural networks.

show abstract

Inferring Point Clouds from Single Monocular Images by Depth Intermediation

Cited by 3 publications

References 30 publications

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images

Can Language Understand Depth?

Contact Info

Product

Resources

About