Depth Completion and Super-Resolution with Arbitrary Scale Factors for Indoor Scenes

Abstract

Depth sensing has improved rapidly in recent years, which allows for structural information to be utilized in various applications, such as virtual reality, scene and object recognition, view synthesis, and 3D reconstruction. Due to the limitations of the current generation of depth sensors, the resolution of depth maps is often still much lower than the resolution of color images. This hinders applications, such as view synthesis or 3D reconstruction, from providing high-quality results. Therefore, super-resolution, which allows for the upscaling of depth maps while still retaining sharpness, has recently drawn much attention in the deep learning community. However, state-of-the-art deep learning methods are typically designed and trained to handle a fixed set of integer-scale factors. Moreover, the raw depth map collected by the depth sensor usually has many depth data missing or misestimated values along the edges and corners of observed objects. In this work, we propose a novel deep learning network for both depth completion and depth super-resolution with arbitrary scale factors. The experimental results on the Middlebury stereo, NYUv2, and Matterport3D datasets demonstrate that the proposed method can outperform state-of-the-art methods.

Single Image Super-Resolution with Arbitrary Magnification Based on High-Frequency Attention Network

“…According to Truong et al [26], a depth map allows for structural information to be utilized in various applications, such as view synthesis or 3D reconstruction. However, the resolution of depth maps is often much lower than the resolution of RGB images due to the limitations of depth sensors.…”

“…To meet the requirement, they proposed arbitrary scale sup resolution (ASSR) that consists of a feature learning module and an arbitrary upscale m ule. According to Truong et al [26], a depth map allows for structural information to utilized in various applications, such as view synthesis or 3D reconstruction. Howev the resolution of depth maps is often much lower than the resolution of RGB images d to the limitations of depth sensors.…”

“…To solve this problem, the low-resolution depth im can be upscaled to target resolution through the depth image super-resolution with ar trary scale factors, as shown in Figure 4. According to Truong et al [26], a depth map allows for structural information to be utilized in various applications, such as view synthesis or 3D reconstruction. However, the resolution of depth maps is often much lower than the resolution of RGB images due to the limitations of depth sensors.…”