ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

Abstract

Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can now impact people's everyday experiences. However, transforming these scene understanding methods to real-world experiences requires additional innovation and development. In this paper we introduce ARKitScenes. It is not only the first RGB-D dataset that is captured with a now widely available depth sensor, but to our best knowledge, it also is the largest indoor scene understanding data released. In addition to the raw and processed data from the mobile device, ARKitScenes includes high resolution depth maps captured using a stationary laser scanner, as well as manually labeled 3D oriented bounding boxes for a large taxonomy of furniture. We further analyze the usefulness of the data for two downstream tasks: 3D object detection and color-guided depth upsampling. We demonstrate that our dataset can help push the boundaries of existing state-of-the-art methods and it introduces new challenges that better represent real-world scenarios. * Equal contribution, alphabetical order.

RGB-D Robotic Pose Estimation For a Servicing Robotic Arm

“…Kinect V1/V2, 11 OAK-D-Lite, 12 and even iPhone's back facing camera satisfy the database requirements. 13 A depth image produced using iPhone camera is shown below in Figure 2. After experimentation with several cameras (iPhone X, OAK-D-Lite, Intel RealSense L515, Kinect V1, Kinect V2), it was decided to use Kinect V2 as it produced the most detailed depth map out of all tested cameras.…”

Bounding Box Disparity: 3D Metrics for Object Detection with Full Degree of Freedom

“…Its elements correspond to the column-index in the matrix of corners. [2,8], [3,8], [1,3], [4,7], [7,5], [6,5], [4,6], [1,4], [2,7], [8,5], [3,6]] (4) [1,2,4], [1,3,4], [5,6,7], [5,6,8], [5,7,8]]…”

“…As 3D object detection gets more popular and new datasets are published [1,2,3], evaluation metrics gain in importance. The most common one is Intersection over Union (IoU).…”

Consistent Direct Time-of-Flight Video Depth Super-Resolution

“…On-device depth estimation is critical in navigation [40], gaming [6], and augmented/virtual reality [3,8]. Previously, various solutions based on stereo/structured-light sensors and indirect time-of-flight sensors (iToF) [4, 34,55] have been proposed.…”

“…Each dToF pixel captures and pre-processes depth information from a local patch in the scene (Sec. 3), leading to high spatial ambiguity when estimating the high-resolution depth maps for downstream tasks [8]. Previous RGB-guided depth completion and super-resolution algorithms either assume high resolution spatial information (e.g.…”