Depth upsampling based on deep edge-aware learning

Wang, Zhihui; Ye, Xinchen; Sun, Baoli; Yang, Jingyu; Xu, Rui; Li, Haojie

doi:10.1016/j.patcog.2020.107274

Cited by 53 publications

(32 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experiments had proved that this kind of quality measures which is similar to human perception is more reasonable. Wang et al [35] proposed a cascaded restoration network, which considered the edge and color information of the input image. Experimental results showed that restoration module including edge information improved the boundaries resolution of recovered depth images.…”

Section: Learning-based Depth Map Sr Methodsmentioning

confidence: 99%

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution

Xian,

Qian,

Zhang

et al. 2020

Preprint

View full text Add to dashboard Cite

Limited by the cost and technology, the resolution of depth map collected by depth camera is often lower than that of its associated RGB camera. Although there have been many researches on RGB image super-resolution (SR), a major problem with depth map super-resolution is that there will be obvious jagged edges and excessive loss of details. To tackle these difficulties, in this work, we propose a multi-scale progressive fusion network for depth map SR, which possess an asymptotic structure to integrate hierarchical features in different domains. Given a low-resolution (LR) depth map and its associated high-resolution (HR) color image, We utilize two different branches to achieve multi-scale feature learning. Next, we propose a stepwise fusion strategy to restore the HR depth map. Finally, a multi-dimensional loss is introduced to constrain clear boundaries and details. Extensive experiments show that our proposed method produces improved results against state-of-the-art methods both qualitatively and quantitatively.

show abstract

Section: Learning-based Depth Map Sr Methodsmentioning

confidence: 99%

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution

Xian,

Qian,

Zhang

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Wen et al [40] used the color information as guidance to infer an initial HR depth map, then proposed a coarse-to-fine networks to progressively optimize the depth map. Wang et al [39] put forward a DSR network to learn a binary map of depth edges and then recovered the HR depth map based on edge-guided filter or cascaded network modules. However, due to treating all depth regions equally without considering depth range variation, there is still room for the above methods to make further improvement.…”

Section: Related Work 21 Cnn-based Depth Super-resolutionmentioning

confidence: 99%

“…Depth SR under Noiseless Cases: To validate the superiority of our method, we first evaluate noiseless cases on Middlebury and NYU datasets, respectively. For Middlebury dataset, we compare with DJF [25], MSG [17], DGDIE [12], DEIN [41], CCFN [40], GSRPT [3], D-SR N [39], which are learning-based methods based on color guidance. Table 1 presents the ×2, ×4, ×8 and ×16 upsampling performance of different methods.…”

Section: Performance Comparisonmentioning

confidence: 99%

Depth Super-Resolution via Deep Controllable Slicing Network

Sun

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Due to the imaging limitation of depth sensors, high-resolution (HR) depth maps are often difficult to be acquired directly, thus effective depth super-resolution (DSR) algorithms are needed to generate HR output from its low-resolution (LR) counterpart. Previous methods treat all depth regions equally without considering different extents of degradation at region-level, and regard DSR under different scales as independent tasks without considering the modeling of different scales, which impede further performance improvement and practical use of DSR. To alleviate these problems, we propose a deep controllable slicing network from a novel perspective. Specifically, our model is to learn a set of slicing branches in a divide-and-conquer manner, parameterized by a distance-aware weighting scheme to adaptively aggregate different depths in an ensemble. Each branch that specifies a depth slice (e.g., the region in some depth range) tends to yield accurate depth recovery. Meanwhile, a scale-controllable module that extracts depth features under different scales is proposed and inserted into the front of slicing network, and enables finely-grained control of the depth restoration results of slicing network with a scale hyper-parameter. Extensive experiments on synthetic and real-world benchmark datasets demonstrate that our method achieves superior performance. CCS CONCEPTS • Computing methodologies → 3D imaging; Reconstruction.

show abstract

“…Tosic et al [40] present a method for learning joint depth and intensity sparse generative models and use joint basic pursuit to infer sparse coefficients. Motivated by the success of DCNN, deep learning-based methods [41]- [47] are also developed. The complete deep primal-dual network in [41] contains a fully convolutional network for an initial HR depth estimation, and a non-local variational primal-dual network for HR depth refinement.…”

Section: ) Color Guided Depth Map Super-resolutionmentioning

confidence: 99%

“…A residual UNet deep network named DepthSR-Net is presented in [46] to infer HR depth map by hierarchical features driven residual learning. A deep edgeaware learning framework in [47] is used to estimate depth edges as reconstruction cues and then two depth restoration modules are used to recover HR depth map. The proposed method in this paper also belongs to the deep learning-based SR category with color guidance.…”

Section: ) Color Guided Depth Map Super-resolutionmentioning

confidence: 99%

Guided Depth Map Super-Resolution Using Recumbent Y Network

Dong

Lin

2020

IEEE Access

View full text Add to dashboard Cite

Low spatial resolution is a well-known problem for depth maps captured by low-cost consumer depth cameras. Depth map super-resolution (SR) can be used to enhance the resolution and improve the quality of depth maps. In this paper, we propose a recumbent Y network (RYNet) to integrate the depth information and intensity information for depth map SR. Specifically, we introduce two weight-shared encoders to respectively learn multi-scale depth and intensity features, and a single decoder to gradually fuse depth information and intensity information for reconstruction. We also design a residual channel attention based atrous spatial pyramid pooling structure to further enrich the feature's scale diversity and exploit the correlations between multi-scale feature channels. Furthermore, the violations of co-occurrence assumption between depth discontinuities and intensity edges will generate texture-transfer and depth-bleeding artifacts. Thus, we propose a spatial attention mechanism to mitigate the artifacts by adaptively learning the spatial relevance between intensity features and depth features and reweighting the intensity features before fusion. Experimental results demonstrate the superiority of the proposed RYNet over several state-of-the-art depth map SR methods. INDEX TERMS Depth map super-resolution, convolutional neural network, UNet network, atrous spatial pyramid pooling, attention mechanism.

show abstract

Depth upsampling based on deep edge-aware learning

Cited by 53 publications

References 16 publications

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution

Depth Super-Resolution via Deep Controllable Slicing Network

Guided Depth Map Super-Resolution Using Recumbent Y Network

Contact Info

Product

Resources

About