Depth map upsampling by self-guided residual interpolation

Konno, Yosuke; Tanaka, Masayuki; Okutomi, Masatoshi; Yanagawa, Yukiko; Kinoshita, Koichi; Kawade, Masato

doi:10.1109/icpr.2016.7899832

Cited by 11 publications

(11 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared the proposed method with the following three method categories: 1) the single-frame depth image upsampling approach including general image upsampling methods: bicubic interpolation, self-guided residual interpolation (SG) [9], edge-guided upsampling (EG) [10], and SRCNN [11]; 2) the intensity-guided depth upsampling approach: guided image filtering (GF) [12], fast global image smoothing (FGI) [15], robust color guided restoration (RCG) [16], and joint local structure and nonlocal low-rank regularization (LN) [17]; and 3) the multi-frame depth image SR approach: robust SR (RSR) [20] based on the BTV model, combination SR (CSR) using sparse and non-sparse priors [23], and multiframe ToF SR (MTSR) [25] based on the TV model. The implementations from the publicly available codes provided by the authors and appropriate parameters were used in our experiments.…”

Section: B Compared Methodsmentioning

confidence: 99%

“…Numerous attempts have been made to overcome the physical limitations of ToF sensors, such as their limited spatial resolution. Algorithms for improving the depth image resolution of ToF sensors can be categorized into three groups, which are described as follows: The first approach is singleframe depth upsampling, which aims to recover a highresolution (HR) image from a single low-resolution (LR) image [9], [10]. Such methods generally use the statistical characteristics inside the depth image or those of the training images.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

2020

View full text Add to dashboard Cite

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L 1 data fidelity with the total variation regularization (TV-L 1) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L 1 model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L 1 data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct highresolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise. INDEX TERMS Depth super-resolution, time-of-flight sensor, multi-frame super-resolution, total variation regularization, L 1 data fidelity.

show abstract

Section: B Compared Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

2020

View full text Add to dashboard Cite

show abstract

“…As the quantitative evaluation metric, we used structural similarity (SSIM) [35] since it can predict human perception of image quality. Similar to Konno et al [36], the standard deviation of Gaussian function in SSIM was set to 4 so that it can evaluate the similarity of semiglobal structure. The higher SSIM value shows a better performance.…”

Section: Methodsmentioning

confidence: 99%

Robust Semiautomatic 2D-to-3D Conversion with Welsch M-Estimator for Data Fidelity

Yuan

et al. 2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Semiautomatic 2D-to-3D conversion plays an important role in generating 3D contents for display. However, most existing methods assume that user scribbles are perfectly correct, and only give acceptable results when user provides accurate labels. To address this problem, Welsch M-estimator data fidelity is used to resist erroneous scribbles. The Welsch M-estimator data fidelity which is able to alleviate the influence of inaccurate scribbles has theoretical guarantee by means of its redescending property. First, the Welsch Mestimator is introduced to measure the fidelity between estimated depth and user provided depth; then local smoothness is built by using color weighted Welsch M-estimator to make neighboring pixels with similar colors have similar depth values. Finally, we solve the problem using generalized iteratively reweighted least squares algorithm. Experiments demonstrate that our method obtains competitive performance in the absence of inaccurate scribbles and outperforms the state of the art both visually and quantitatively in the presence of inaccurate scribbles.

show abstract

“…Structural similarity (SSIM) [38] is used for performance evaluation since it can predict human perception of image quality. The standard deviation of SSIM in the experiments is set to 4 so as to evaluate the similarity of semi-global structure [39].…”

Section: Methodsmentioning

confidence: 99%

Robust semi-automatic 2D-to-3D image conversion via residual-driven optimization

Yuan

2018

J Image Video Proc.

View full text Add to dashboard Cite

Semi-automatic 2D-to-3D conversion provides a cost-effective solution to the problem of 3D content shortage. The performance of most methods degrades significantly when cross-boundary scribbles are present due to their inability to remove unwanted input. To address this problem, a residual-driven energy function is proposed to remove unwanted input introduced by cross-boundary scribbles while preserving expected user input. Firstly, confidence of user input is computed from residuals between the estimation and user-specified depth values, and it is applied to the data fidelity term. Secondly, the residual-driven optimization is performed to estimate dense depth from user scribbles. The procedure is repeated until a maximum number of iterations is exceeded. Input confidence based on residuals avoids the propagation of unwanted scribbles and thus enables to generate high-quality depth even with cross-boundary input. Experimental results demonstrate that the proposed method removes unwanted scribbles successfully while preserving expected input, and it outperforms the state-of-the-art when presented with cross-boundary scribbles.

show abstract

Depth map upsampling by self-guided residual interpolation

Cited by 11 publications

References 26 publications

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

Robust Semiautomatic 2D-to-3D Conversion with Welsch M-Estimator for Data Fidelity

Robust semi-automatic 2D-to-3D image conversion via residual-driven optimization

Contact Info

Product

Resources

About