Plane-Based Optimization of Geometry and Texture for RGB-D Reconstruction of Indoor Scenes

Wang, Chao; Guo, Xiaohu

doi:10.1109/3dv.2018.00067

Cited by 15 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared PonNet with a baseline method: classic RGBD plane detection algorithm [28] (available at this URL 3 ) for free area detection. We also performed several ablation studies with different PonNet types using no attention, i.e., ResNet-50 only (type1) and ResNet-18 only (type2) and a single attention branch [7] (type3).…”

Section: Quantitative Resultsmentioning

confidence: 99%

Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

Magassouba¹,

Sugiura²,

Nakayama³

et al. 2021

Preprint

View full text Add to dashboard Cite

Placing objects is a fundamental task for domestic service robots (DSRs). Thus, inferring the collision-risk before a placing motion is crucial for achieving the requested task. This problem is particularly challenging because it is necessary to predict what happens if an object is placed in a cluttered designated area. We show that a rulebased approach that uses plane detection, to detect free areas, performs poorly. To address this, we develop PonNet, which has multimodal attention branches and a self-attention mechanism to predict damaging collisions, based on RGBD images. Our method can visualize the risk of damaging collisions, which is convenient because it enables the user to understand the risk. For this purpose, we build and publish an original dataset that contains 12,000 photo-realistic images of specific placing areas, with daily life objects, in home environments. The experimental results show that our approach improves accuracy compared with the baseline methods.

show abstract

Section: Quantitative Resultsmentioning

confidence: 99%

Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

Magassouba¹,

Sugiura²,

Nakayama³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, texturing such abstracted building models has drawn less attention. Several methods take into account the generation of texture maps while conducting a lightweight geometric reconstruction [Garcia-Dorado et al 2013;Huang et al 2017;Maier et al 2017;Sinha et al 2008;Wang and Guo 2018]. For example, Garcia-Dorado et al [2013] computes a per-building volumetric proxy, then uses a surface graph-cut method to stitch aerial images and yield a visually coherent texture map.…”

Section: Structure-aware Scene Reconstruction and Texturingmentioning

confidence: 99%

TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models

Xiong,

Zhang,

Peng

et al. 2023

ACM Trans. Graph.

View full text Add to dashboard Cite

Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex , the first automatic texture mapping framework to generate a photorealistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort.

show abstract

“…Different from the above-mentioned works, some recent research has attempted to optimize texture and geometry jointly. Wang et al [23] used planar primitives to partition a model and jointly optimized plane parameters, camera poses, texture and geometry using photometric consistency and planar constraints. However, this method relies on plane priors, and is not suitable for complex nonplanar objects.…”

Section: Geometry and Texture Optimizationmentioning

confidence: 99%

Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering

Zhang,

Wan,

Liao

2022

Preprint

View full text Add to dashboard Cite

Due to inevitable noises introduced during scanning and quantization, 3D reconstruction via RGB-D sensors suffers from errors both in geometry and texture, leading to artifacts such as camera drifting, mesh distortion, texture ghosting, and blurriness. Given an imperfect reconstructed 3D model, most previous methods have focused on the refinement of either geometry, texture, or camera pose. Or different optimization schemes and objectives for optimizing each component have been used in previous joint optimization methods, forming a complicated system. In this paper, we propose a novel optimization approach based on differentiable rendering, which integrates the optimization of camera pose, geometry, and texture into a unified framework by enforcing consistency between the rendered results and the corresponding RGB-D inputs. Based on the unified framework, we introduce a joint optimization approach to fully exploit the inter-relationships between geometry, texture, and camera pose, and describe an adaptive interleaving strategy to improve optimization stability and efficiency. Using differentiable rendering, an image-level adversarial loss is applied to further improve the 3D model, making it more photorealistic. Experiments on synthetic and real data using quantitative and qualitative evaluation demonstrated the superiority of our approach in recovering both fine-scale geometry and high-fidelity texture.

show abstract

Plane-Based Optimization of Geometry and Texture for RGB-D Reconstruction of Indoor Scenes

Cited by 15 publications

References 25 publications

Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations

TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models

Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering

Contact Info

Product

Resources

About