RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

Wald, Johanna; Avetisyan, Armen; Navab, Nassir; Tombari, Federico; Nießner, Matthias

doi:10.1109/iccv.2019.00775

Cited by 95 publications

(75 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we present different experiments to analyse the main aspects of our proposal. First, we evaluate our 3D semantic segmentation network on 3RScan (Wald et al 2019), see Sect. 5.1 to validate the quality of our underlying features (see Sect.…”

Section: Discussionmentioning

confidence: 99%

“…Gay et al (2018) on the other hand, feed object features extracted from a continuous image sequence into a recurrent neural network. They operate in 2.5D on a static setup while 3RScan (Wald et al 2019) has dynamically changing scenes which enables new, challenging tasks such as the newly introduced 2D-3D scene retrieval.…”

Section: D Object Context and Scene Layoutmentioning

confidence: 99%

“…3D Scene Retrieval Visual retrieval has a long history in computer vision, and is often embedded in other tasks such as object detection, object or scene alignment or camera pose estimation (Gálvez-López and Tardós 2011; Torii et al 2015;Glocker et al 2015;Anoosheh et al 2019;Arandjelović et al 2016;Deng et al 2016;Lu et al 2016b). Retrieving a source given some query data becomes most challenging when they do not share the same domain (Dahnert et al 2019;Abdul-Rashid et al 2018Avetisyan et al 2019Avetisyan et al , 2020 including natural scene changes (Wald et al 2020b(Wald et al , 2019. An extensive literature exists on retrieval of CAD models given an image (Izadinia et al 2017;Sun et al 2018) or a 3D scene (Avetisyan et al 2019(Avetisyan et al , 2020.…”

Section: D Object Context and Scene Layoutmentioning

confidence: 99%

“…The method proposed in this work is trained and evaluated on 3DSSG which is based on 3RScan (Wald et al 2019), a collection of every day 3D indoor scenes with 1482 sequences with semantically segmented 3D models. It features approximately 450 unique, diverse indoor environments, captured over a long period of time with change annotations (Wald et al 2019). Unlike any other dataset, this allows reasoning about object instances and their changes but also about their relationships.…”

Section: D Semantic Scene Graphsmentioning

confidence: 99%

“…3.2). More details about the annotation procedure of 3RScan and 3DSSG can be found in Wald et al (2019) and Wald et al (2020a) respectively.…”

Section: D Semantic Scene Graphsmentioning

confidence: 99%

See 4 more Smart Citations

Learning 3D Semantic Scene Graphs with Instance Embeddings

2022

Self Cite

View full text Add to dashboard Cite

A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG, a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: D Object Context and Scene Layoutmentioning

confidence: 99%

Section: D Object Context and Scene Layoutmentioning

confidence: 99%

Section: D Semantic Scene Graphsmentioning

confidence: 99%

“…3.2). More details about the annotation procedure of 3RScan and 3DSSG can be found in Wald et al (2019) and Wald et al (2020a) respectively.…”

Section: D Semantic Scene Graphsmentioning

confidence: 99%

See 3 more Smart Citations

Learning 3D Semantic Scene Graphs with Instance Embeddings

2022

Self Cite

View full text Add to dashboard Cite

show abstract

re-OBJ: Jointly Learning the Foreground and Background for Object Instance Re-identification

Bansal

James

Bue

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Conventional approaches to object instance re-identification rely on matching appearances of the target objects among a set of frames. However, learning appearances of the objects alone might fail when there are multiple objects with similar appearance or multiple instances of same object class present in the scene. This paper proposes that partial observations of the background can be utilized to aid in the object reidentification task for a rigid scene, especially a rigid environment with a lot of reoccurring identical models of objects. Using an extension to the Mask R-CNN architecture, we learn to encode the important and distinct information in the background jointly with the foreground relevant to rigid real-world scenarios such as an indoor environment where objects are static and the camera moves around the scene. We demonstrate the effectiveness of our joint visual feature in the re-identification of objects in the ScanNet dataset and show a relative improvement of around 28.25% in the rank-1 accuracy over the deepSort method.

show abstract

Learning 3D Part Assembly from a Single Image

Shao

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Autonomous assembly is a crucial capability for robots in many applications. For this task, several problems such as obstacle avoidance, motion planning, and actuator control have been extensively studied in robotics. However, when it comes to task specification, the space of possibilities remains underexplored. Towards this end, we introduce a novel problem, single-image-guided 3D part assembly, along with a learningbased solution. We study this problem in the setting of furniture assembly from a given complete set of parts and a single image depicting the entire assembled object. Multiple challenges exist in this setting, including handling ambiguity among parts (e.g., slats in a chair back and leg stretchers) and 3D pose prediction for parts and part subassemblies, whether visible or occluded. We address these issues by proposing a two-module pipeline that leverages strong 2D-3D correspondences and assembly-oriented graph message-passing to infer part relationships. In experiments with a PartNet-based synthetic benchmark, we demonstrate the effectiveness of our framework as compared with three baseline approaches.

show abstract

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

Cited by 95 publications

References 28 publications

Learning 3D Semantic Scene Graphs with Instance Embeddings

Learning 3D Semantic Scene Graphs with Instance Embeddings

re-OBJ: Jointly Learning the Foreground and Background for Object Instance Re-identification

Learning 3D Part Assembly from a Single Image

Contact Info

Product

Resources

About