SceneCode: Monocular Dense Semantic Reconstruction Using Learned Encoded Scene Representations

Zhi, Shuaifeng; Bloesch, Michael; Leutenegger, Stefan; Davison, Andrew J.

doi:10.1109/cvpr.2019.01205

Cited by 67 publications

(36 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A sample reconstruction is shown in Figure 5. With computer vision research evolving rapidly, we believe constructing geometry proxies from video input will become even more robust and easily accessible [3,69].…”

Section: Geometry Reconstructionmentioning

confidence: 99%

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1: Given a natural sound in a real-world room that is recorded using a cellphone microphone (left), we estimate the acoustic material properties and the frequency equalization of the room using a novel deep learning approach (middle). We use the estimated acoustic material properties for generating plausible sound effects in the virtual model of the room (right). Our approach is general and robust, and works well with commodity devices.Abstract-We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

show abstract

Section: Geometry Reconstructionmentioning

confidence: 99%

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…Preliminary work exists towards joint geometric and semantic SLAM (e.g. [5]); yet, these systems are fairly limited in terms of accuracy and scaling. Instead, the majority of state-of-the-art work relies on a sequential geometric reconstruction and frame-wise labelling, followed by semantic fusion.…”

Section: Related Workmentioning

confidence: 99%

Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAM

Landgraf

Falck

Bloesch

et al. 2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

Generally capable Spatial AI systems must build persistent scene representations where geometric models are combined with meaningful semantic labels. The many approaches to labelling scenes can be divided into two clear groups: view-based which estimate labels from the input viewwise data and then incrementally fuse them into the scene model as it is built; and map-based which label the generated scene model. However, there has so far been no attempt to quantitatively compare view-based and map-based labelling. Here, we present an experimental framework and comparison which uses real-time height map fusion as an accessible platform for a fair comparison, opening up the route to further systematic research in this area.

show abstract

“…Hu et al [51] finally extend this approach to a complete SLAM framework that optimises a larger scale graph over many frames and multiple complex-shaped objects of different classes (e.g., chairs, tables) as well as latent shape representations for each object instance in parallel. Note that low-dimensional latent representations for modelling 3D geometry have also been utilised by Bloesch et al [52] and Zhi et al [53]. More specifically, they rely on photometric consistency to optimise codes in each keyframe that generate depth maps using a deconvolutional architecture.…”

Section: Review Of Current Slam Systems and Their Evolution Into Spatmentioning

confidence: 99%

Representations and Benchmarking of Modern Visual SLAM Systems

Cao

Kneip

2020

Sensors

View full text Add to dashboard Cite

Simultaneous Localisation And Mapping (SLAM) has long been recognised as a core problem to be solved within countless emerging mobile applications that require intelligent interaction or navigation in an environment. Classical solutions to the problem primarily aim at localisation and reconstruction of a geometric 3D model of the scene. More recently, the community increasingly investigates the development of Spatial Artificial Intelligence (Spatial AI), an evolutionary paradigm pursuing a simultaneous recovery of object-level composition and semantic annotations of the recovered 3D model. Several interesting approaches have already been presented, producing object-level maps with both geometric and semantic properties rather than just accurate and robust localisation performance. As such, they require much broader ground truth information for validation purposes. We discuss the structure of the representations and optimisation problems involved in Spatial AI, and propose new synthetic datasets that, for the first time, include accurate ground truth information about the scene composition as well as individual object shapes and poses. We furthermore propose evaluation metrics for all aspects of such joint geometric-semantic representations and apply them to a new semantic SLAM framework. It is our hope that the introduction of these datasets and proper evaluation metrics will be instrumental in the evaluation of current and future Spatial AI systems and as such contribute substantially to the overall research progress on this important topic.

show abstract

SceneCode: Monocular Dense Semantic Reconstruction Using Learned Encoded Scene Representations

Cited by 67 publications

References 29 publications

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAM

Representations and Benchmarking of Modern Visual SLAM Systems

Contact Info

Product

Resources

About