Multi-frame Super-resolution for Time-of-flight Imaging

Li, Fengqiang; Ruiz, Pablo; Cossairt, Oliver; Katsaggelos, Aggelos K.

doi:10.1109/icassp.2019.8683042

Cited by 6 publications

(18 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared the proposed method with the following three method categories: 1) the single-frame depth image upsampling approach including general image upsampling methods: bicubic interpolation, self-guided residual interpolation (SG) [9], edge-guided upsampling (EG) [10], and SRCNN [11]; 2) the intensity-guided depth upsampling approach: guided image filtering (GF) [12], fast global image smoothing (FGI) [15], robust color guided restoration (RCG) [16], and joint local structure and nonlocal low-rank regularization (LN) [17]; and 3) the multi-frame depth image SR approach: robust SR (RSR) [20] based on the BTV model, combination SR (CSR) using sparse and non-sparse priors [23], and multiframe ToF SR (MTSR) [25] based on the TV model. The implementations from the publicly available codes provided by the authors and appropriate parameters were used in our experiments.…”

Section: B Compared Methodsmentioning

confidence: 99%

“…However, the source code of LN [17] was executed only for the downsampling degradation on the Middlebury dataset. Experiments were conducted on the Tikhonov-L 2 and TV-L 2 models implemented with our scheme (denoted by "ours"), as proposed in Lidarboost [24] and MTSR [25]. We selected the regularization parameters that generated the highest PSNR values for all of the testing images.…”

Section: B Compared Methodsmentioning

confidence: 99%

“…Ng et al formulated the multi-frame SR reconstruction method by minimizing the TV-L 2 objective function [21], which was recently applied to multi-frame depth SR [25]. This method is effective in suppressing ringing artifacts and preventing sharp edges from oversmoothing.…”

Section: B Multi-frame Depth Srmentioning

confidence: 99%

“…In terms of increasing the resolution of the ToF sensor, a technique using multi-frame depth images has been developed by extending the classical SR method for existing cameras [24]. Recently, a method based on the TV prior model has been proposed to preserve the overall sharpness of depth images [25]. However, the multi-frame depth SR techniques for ToF sensors presented in [24] and [25] do not reflect the depth image characteristics, and do not consider problems such as oversmoothing and noise amplification.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, a method based on the TV prior model has been proposed to preserve the overall sharpness of depth images [25]. However, the multi-frame depth SR techniques for ToF sensors presented in [24] and [25] do not reflect the depth image characteristics, and do not consider problems such as oversmoothing and noise amplification. Therefore, we propose a method based on the L 1 data fidelity with the TV regularization (TV-L 1 ) model to overcome the limitations of other methods in terms of preserving the sharp depth discontinuities and suppressing the noise amplification.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

2020

View full text Add to dashboard Cite

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L 1 data fidelity with the total variation regularization (TV-L 1) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L 1 model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L 1 data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct highresolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise. INDEX TERMS Depth super-resolution, time-of-flight sensor, multi-frame super-resolution, total variation regularization, L 1 data fidelity.

show abstract

Section: B Compared Methodsmentioning

confidence: 99%

Section: B Compared Methodsmentioning

confidence: 99%

Section: B Multi-frame Depth Srmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

2020

View full text Add to dashboard Cite

show abstract

Raw Data Processing for Practical Time-of-Flight Super-Resolution

Conde

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The relatively low resolution of Time-of-Flight (ToF) cameras, together with high power consumption and motion artifacts due to long exposure times, have kept ToF sensors away from classical lidar application fields, such as mobile robotics and autonomous driving. In this paper we note that while attempting to address the last two issues, e. g., via burst mode, the lateral resolution can be effectively increased. Differently from prior approaches, we propose a strippeddown modular super-resolution framework that operates in the raw data domain. Experimental validation using synthetic raw data from standard data sets and real raw data from a PMD ToF sensor confirmed the feasibility of the proposed pipeline.

show abstract