Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning

Feng, Qiaojun; Atanasov, Nikolay

doi:10.1109/icra48506.2021.9561337

Cited by 6 publications

(17 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…6). To obtain a 2D semantic segmentation image from the mesh, we use the differentiable semantic renderer ρ S introduced in (7). Both the vertex spatial coordinates V and the semantic features C can affect the rendered 2D semantic segmentation image ρ S (M).…”

Section: Semantic Mesh Reconstructionmentioning

confidence: 99%

“…Both the vertex spatial coordinates V and the semantic features C can affect the rendered 2D semantic segmentation image ρ S (M). Hence, by optimizing the semantic loss in (7), we can refine both the semantic features and the geometric structure of the mesh.…”

Section: Semantic Mesh Reconstructionmentioning

confidence: 99%

“…5) Semantic Loss Function: Finally, we discuss the choice of a semantic loss function S in (7). Instead of the Dice loss in (7), three other semantic loss functions may be considered.…”

Section: E Ablation Studiesmentioning

confidence: 99%

“…where, as in (7), | • | sums up all the absolute values of the elements. In Table III, we see that the Jaccard loss in (22) leads to good segmentation performance, outperforming the Dice loss in (7) for some categories. The Cross Entropy and the Focal losses are not as good.…”

Section: E Ablation Studiesmentioning

confidence: 99%

“…This paper is an extension of our 2021 IEEE ICRA conference paper [7] on mesh reconstruction from aerial RGB images and sparse depth measurements. We propose a joint 2D-3D learning method for metric-semantic mesh reconstruction using RGB images and sparse depth measurements.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning

Feng¹,

Atanasov²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper considers outdoor terrain mapping using RGB images obtained from an aerial vehicle. While feature-based localization and mapping techniques deliver real-time vehicle odometry and sparse keypoint depth reconstruction, a dense model of the environment geometry and semantics (vegetation, buildings, etc.) is usually recovered offline with significant computation and storage. This paper develops a joint 2D-3D learning approach to reconstruct a local metric-semantic mesh at each camera keyframe maintained by a visual odometry algorithm. Given the estimated camera trajectory, the local meshes can be assembled into a global environment model to capture the terrain topology and semantics during online operation. A local mesh is reconstructed using an initialization and refinement stage. In the initialization stage, we estimate the mesh vertex elevation by solving a least squares problem relating the vertex barycentric coordinates to the sparse keypoint depth measurements. In the refinement stage, we associate 2D image and semantic features with the 3D mesh vertices using camera projection and apply graph convolution to refine the mesh vertex spatial coordinates and semantic features based on joint 2D and 3D supervision. Quantitative and qualitative evaluation using real aerial images show the potential of our method to support environmental monitoring and surveillance applications.

show abstract

Section: Semantic Mesh Reconstructionmentioning

confidence: 99%

Section: Semantic Mesh Reconstructionmentioning

confidence: 99%

“…5) Semantic Loss Function: Finally, we discuss the choice of a semantic loss function S in (7). Instead of the Dice loss in (7), three other semantic loss functions may be considered.…”

Section: E Ablation Studiesmentioning

confidence: 99%

Section: E Ablation Studiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning

Feng¹,

Atanasov²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Learning to sculpt neural cityscapes

Zhu,

Wang,

Hogg

et al. 2024

Vis Comput

View full text Add to dashboard Cite

We introduce a system that learns to sculpt 3D models of massive urban environments. The majority of humans live their lives in urban environments, using detailed virtual models for applications as diverse as virtual worlds, special effects, and urban planning. Generating such 3D models from exemplars manually is time-consuming, while 3D deep learning approaches have high memory costs. In this paper, we present a technique for training 2D neural networks to repeatedly sculpt a plane into a large-scale 3D urban environment. An initial coarse depth map is created by a GAN model, from which we refine 3D normal and depth using an image translation network regularized by a linear system. The networks are trained using real-world data to allow generative synthesis of meshes at scale. We exploit sculpting from multiple viewpoints to generate a highly detailed, concave, and water-tight 3D mesh. We show cityscapes at scales of $$100 \times 1600$$ 100 × 1600 meters with more than 2 million triangles, and demonstrate that our results are objectively and subjectively similar to our exemplars.

show abstract

Survey on Monocular Depth Estimation for Unmanned Aerial Vehicles using Deep Learning

Florea

Nedevschi

2022

2022 IEEE 18th International Conference on Intelligent Computer Communication and Processing (ICCP)

View full text Add to dashboard Cite

Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning

Cited by 6 publications

References 18 publications

TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning

TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning

Learning to sculpt neural cityscapes

Survey on Monocular Depth Estimation for Unmanned Aerial Vehicles using Deep Learning

Contact Info

Product

Resources

About