SmartAnnotator An Interactive Tool for Annotating Indoor RGBD Images

Abstract

RGBD images with high quality annotations, both in the form of geometric (i.e., segmentation) and structural (i.e., how do the segments mutually relate in 3D) information, provide valuable priors for a diverse range of applications in scene understanding and image manipulation. While it is now simple to acquire RGBD images, annotating them, automatically or manually, remains challenging. We present SMARTANNOTATOR, an interactive system to facilitate annotating raw RGBD images. The system performs the tedious tasks of grouping pixels, creating potential abstracted cuboids, inferring object interactions in 3D, and generates an ordered list of hypotheses. The user simply has to flip through the suggestions for segment labels, finalize a selection, and the system updates the remaining hypotheses. As annotations are finalized, the process becomes simpler with fewer ambiguities to resolve. Moreover, as more scenes are annotated, the system makes better suggestions based on the structural and geometric priors learned from previous annotation sessions. We test the system on a large number of indoor scenes across different users and experimental settings, validate the results on existing benchmark datasets, and report significant improvements over low-level annotation alternatives. (Code and benchmark datasets are publicly available on the project page.)

On support relations and semantic scene graphs

“…Physical relations between objects to help image or scene understanding have been investigated in [9], [10], [15], [16], and [17]. Pixel-wise segmentation and 3D volumetric estimation are two major methods for this task.…”

A Mobile System for Scene Monitoring and Object Retrieval

“…The problem is that a depth-map image is not as rich in information as a color image. Wong et al [18] proposes a method where the rgbd images are progressively annotated by a user to improve the retrieval. While this can work for some applications it is very impractical for monitoring applications where large spaces are surveyed that contain a large number of objects While a single depth-map image may not be very powerful, multiple depth-maps can be fused together by geometrically aligning their corresponding point clouds [19] and performing the retrieval in geometric space.…”

Scene object recognition for mobile robots through Semantic Knowledge and Probabilistic Graphical Models

“…Therefore, a significant, growing body of current research aiming to overcome this issue is considering contextual information of the scene objects in addition to their usually employed individual features, and a number of applications dealing with this source of information have came out, e.g. Wong et al (2015) or Ruiz-Sarmiento, J.R. et al (2015b). Some works have attempted to exploit this information by providing ad-hoc or preliminary solutions, like in Mekhalfi et al (2015), where the co-occurrence of objects appearing in distinct types of rooms are implicitly modelled.…”