VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion

Stier, Noah; Rich, Alexander; Sen, Pratima; Höllerer, Tobias

doi:10.1109/3dv53792.2021.00042

Cited by 43 publications

(15 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then NeuralRecon (Sun et al, 2021) improves the efficiency by doing this within a local window and then fusing the prediction together using a GRU module. Recently, to improve the accuracy of the reconstruction, some methods also introduce transformers to do the fusion of 2D features from different views (Bozic et al, 2021;Stier et al, 2021). However, their transformers are all limited in 2D space and used to process 2D features, which is not straightforward in the 3D reconstruction task.…”

Section: Related Workmentioning

confidence: 99%

“…Limited by the high resource-consuming of the multi-head attention, most of the previous works related to 3D transformers are only carefully performed on resource-saving feature processing, e.g., the one-off straightforward feature mapping without any downsampling or upsampling (Wang et al, 2021a), where the size of feature volumes remains unchanged, or the top-down tasks with only downsampling (Mao et al, 2021), where the size of feature volumes is reduced gradually. In 3D reconstruction, however, a top-down-bottom-up structure is more reasonable for feature extraction and prediction generation, as in most of the 3D-CNN-based structures (Murez et al, 2020;Sun et al, 2021;Stier et al, 2021). So in this work, we design the first 3D transformer based top-down-bottomup structure, as is shown in Figure 3.…”

Section: Sdf 3d Transformermentioning

confidence: 99%

“…To supervise the averaging weights W l i , we use the occupancy in the back-projection following Stier et al (2021). Intuitively, when the feature is back-projected from a 2D image to the 3D space along the camera ray using multiple depth values, we hope the voxels close to the mesh surface have bigger weights in the fusion.…”

Section: Loss Functionmentioning

confidence: 99%

“…Inspired by recent successes of vision transformer (Vaswani et al, 2017;Dosovitskiy et al, 2020), some approaches (Bozic et al, 2021;Stier et al, 2021) have adopted this structure in 3D recon- struction, but their usages are all limited to fusing the 2D features from different views while the aggregation of the 3D feature volumes is still performed by the 3D CNN. In this paper, we claim that the aggregation of 3D feature volume is also critical, and the evolution from 3D CNN to 3D multi-head attention could further improve both the accuracy and completeness of the reconstruction.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

Yuan¹,

Gu²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Monocular scene reconstruction from posed images is challenging due to the complexity of a large environment. Recent volumetric methods learn to directly predict the TSDF volume and have demonstrated promising results in this task. However, most methods focus on how to extract and fuse the 2D features to a 3D feature volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose an SDF transformer network, which replaces the role of 3D CNN for better 3D feature aggregation. To reduce the explosive computation complexity of the 3D multi-head attention, we propose a sparse window attention module, where the attention is only calculated between the non-empty voxels within a local window. Then a top-down-bottom-up 3D attention network is built for 3D feature aggregation, where a dilate-attention structure is proposed to prevent geometry degeneration, and two global modules are employed to equip with global receptive fields. The experiments on multiple datasets show that this 3D transformer network generates a more accurate and complete reconstruction, which outperforms previous methods by a large margin. Remarkably, the mesh accuracy is improved by 41.8%, and the mesh completeness is improved by 25.3% on the ScanNet dataset. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Sdf 3d Transformermentioning

confidence: 99%

Section: Loss Functionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

Yuan¹,

Gu²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…NeuralRecon [31] reconstructs surfaces sequentially from video fragments as TSDF volumes by performing feature fusion from previous fragments via recurrent units. Atlas [14] aggregates image features over an entire sequence to predict a globally consistent TSDF volume and semantic labels with a 3DNN, while others [2,28] rely on transformers for feature fusion. While the above methods require direct SDF supervision at training time, inference only requires posed RGB video.…”

Section: Related Workmentioning

confidence: 99%

GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction

Wang¹,

Tymoteusz²

2022

Preprint

View full text Add to dashboard Cite

Figure 1: Given an input RGB-D sequence, GO-Surf obtains a high quality 3D surface reconstruction by direct optimization of a multi-resolution feature grid and signed distance value and colour prediction. We formulate a new smoothness prior on the signed distance values that leads to improved hole filling and smoothness properties, while preserving details. Our optimization is ×60 times faster than MLP-based methods.

show abstract

TMSDNet: Transformer with multi‐scale dense network for single and multi‐view 3D reconstruction

Zhu,

Yao,

Zhang

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Abstract3D reconstruction is a long‐standing problem. Recently, a number of studies have emerged that utilize transformers for 3D reconstruction, and these approaches have demonstrated strong performance. However, transformer‐based 3D reconstruction methods tend to establish the transformation relationship between the 2D image and the 3D voxel space directly using transformers or rely solely on the powerful feature extraction capabilities of transformers. They ignore the crucial role played by deep multi‐scale representation of the object in the voxel feature domain, which can provide extensive global shape and local detail information about the object in a multi‐scale manner. In this article, we propose a novel framework TMSDNet (transformer with multi‐scale dense network) for single‐view and multi‐view 3D reconstruction with transformer to solve this problem. Based on our well‐designed combined‐transformer Block, which is canonical encoder–decoder architecture, voxel features with spatial order can be extracted from the input image, which are used to further extract multi‐scale global features in parallel using a multi‐scale residual attention module. Furthermore, a residual dense attention block is introduced for deep local features extraction and adaptive fusion. Finally, the reconstructed objects are produced with the voxel reconstruction block. Experiment results on the benchmarks such as ShapeNet and Pix3D datasets demonstrate that TMSDNet outperforms the existing state‐of‐the‐art reconstruction methods substantially.

show abstract

VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion

Cited by 43 publications

References 39 publications

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction

TMSDNet: Transformer with multi‐scale dense network for single and multi‐view 3D reconstruction

Contact Info

Product

Resources

About