Nonlocal Tensor Completion for Multitemporal Remotely Sensed Images’ Inpainting

Ji, Teng-Yu; Yokoya, Naoto; Zhu, Xiao Xiang; Huang, Ting‐Zhu

doi:10.1109/tgrs.2018.2790262

Cited by 77 publications

(31 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We present our experiments and comparisons of different cloud removal methods: Our proposed STGCN method, the non-local low-rank tensor completion method (NL-LRTC) [16], the recent spatial-temporal-spectral based cloud removal algorithm via CNN (STSCNN) [36], and the very recent temporal-based cloud removal network (CRN) [41]. A partial convolution-based in-painting technology for irregular holes (Pconv) [48] was also executed for quantitative and visual comparison.…”

Section: Cloud Removal Resultsmentioning

confidence: 99%

“…Cheng et al adopted an image in-painting technology based on non-local total variation, in which multi-band data were utilized to achieve spectral coherence. Non-local correlation in the spatial domain and low-rankness in the spatial-temporal-temporal domain were considered separately in reference [16] to reconstruct missing pixels. Recently, He et al [17] designed a new low-rank tensor decomposition method and a total variation model for missing information reconstruction.…”

Section: Of 19mentioning

confidence: 99%

See 1 more Smart Citation

Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images

Dai

Zhang

2020

Remote Sensing

View full text Add to dashboard Cite

Pixels of clouds and cloud shadows in a remote sensing image impact image quality, image interpretation, and subsequent applications. In this paper, we propose a novel cloud removal method based on deep learning that automatically reconstructs the invalid pixels with the auxiliary information from multi-temporal images. Our method’s innovation lies in its feature extraction and loss functions, which reside in a novel gated convolutional network (GCN) instead of a series of common convolutions. It takes the current cloudy image, a recent cloudless image, and the mask of clouds as input, without any requirements of external training samples, to realize a self-training process with clean pixels in the bi-temporal images as natural training samples. In our feature extraction, gated convolutional layers, for the first time, are introduced to discriminate cloudy pixels from clean pixels, which make up for a common convolution layer’s lack of the ability to discriminate. Our multi-level constrained joint loss function, which consists of an image-level loss, a feature-level loss, and a total variation loss, can achieve local and global consistency both in shallow and deep levels of features. The total variation loss is introduced into the deep-learning-based cloud removal task for the first time to eliminate the color and texture discontinuity around cloud outlines needing repair. On the WHU cloud dataset with diverse land cover scenes and different imaging conditions, our experimental results demonstrated that our method consistently reconstructed the cloud and cloud shadow pixels in various remote sensing images and outperformed several mainstream deep-learning-based methods and a conventional method for every indicator by a large margin.

show abstract

Section: Cloud Removal Resultsmentioning

confidence: 99%

Section: Of 19mentioning

confidence: 99%

Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images

Dai

Zhang

2020

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Ng et al [34] and Cheng et al [35] designed adaptive weighted tensor completion methods to recover the cloud-contaminated pixels. Ji et al [36] suggested a nonlocal tensor completion method to restore the missing area. Chen et al [37] proposed a low-rank sparsity decomposition method regularized by total variation, called TVLRSDC, which could remove the cloud and cloud shadow simultaneously.…”

Section: Temporal Information-based Methodsmentioning

confidence: 99%

Thick Cloud Removal of Remote Sensing Images Using Temporal Smoothness and Sparsity Regularized Tensor Optimization

Duan

Pan

2020

Remote Sensing

View full text Add to dashboard Cite

In remote sensing images, the presence of thick cloud accompanying shadow can affect the quality of subsequent processing and limit the scenarios of application. Hence, to make good use of such images, it is indispensable to remove the thick cloud and cloud shadow as well as recover the cloud-contaminated pixels. Generally, the thick cloud and cloud shadow element are not only sparse but also smooth along the spatial horizontal and vertical direction, while the clean element is smooth along the temporal direction. Guided by the above insight, a novel thick cloud removal method for remote sensing images based on temporal smoothness and sparsity regularized tensor optimization (TSSTO) is proposed in this paper. Firstly, the sparsity norm is utilized to boost the sparsity of the cloud and cloud shadow element, and unidirectional total variation (UTV) regularizers are applied to ensure the smoothness in different directions. Then, through thresholding, the cloud mask and the cloud shadow mask can be acquired and used to guide the substitution. Finally, the reference image is selected to reconstruct details of the repairing area. A series of experiments are conducted both on simulated and real cloud-contaminated images from different sensors and with different resolutions, and the results demonstrate the potential of the proposed TSSTO method for removing cloud and cloud shadow from both qualitative and quantitative viewpoints.

show abstract

“…Following [49][50][51], we use lower-case letters for vectors, e.g., a; upper-case letters for matrices, e.g., A; and calligraphic letters for tensors, e.g., A. An N -mode tensor is defined as X ∈ R I1×I2×···×I N , and x i1,i2,··· ,i N denotes its (i 1 , i 2 , · · · , i N )-th component.…”

Section: Fmentioning

confidence: 99%

FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors

Jiang

Huang

Zhao

et al. 2019

IEEE Trans. on Image Process.

Self Cite

152

View full text Add to dashboard Cite

Rain streaks removal is an important issue in outdoor vision systems and has recently been investigated extensively. In this paper, we propose a novel video rain streak removal approach FastDeRain, which fully considers the discriminative characteristics of rain streaks and the clean video in the gradient domain. Specifically, on the one hand, rain streaks are sparse and smooth along the direction of the raindrops, whereas on the other hand, clean videos exhibit piecewise smoothness along the rain-perpendicular direction and continuity along the temporal direction. Theses smoothness and continuity results in the sparse distribution in the different directional gradient domain, respectively. Thus, we minimize 1) the 1 norm to enhance the sparsity of the underlying rain streaks, 2) two 1 norm of unidirectional Total Variation (TV) regularizers to guarantee the anisotropic spatial smoothness, and 3) an 1 norm of the time-directional difference operator to characterize the temporal continuity. A split augmented Lagrangian shrinkage algorithm (SALSA) based algorithm is designed to solve the proposed minimization model. Experiments conducted on synthetic and real data demonstrate the effectiveness and efficiency of the proposed method. According to comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods, especially on account of the running time. The code of FastDeRain can be downloaded at https://github.com/TaiXiangJiang/FastDeRain. Index Terms-video rain streak removal, unidirectional total variation, split augmented Lagrangian shrinkage algorithm (SALSA) .

show abstract

Nonlocal Tensor Completion for Multitemporal Remotely Sensed Images’ Inpainting

Cited by 77 publications

References 66 publications

Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images

Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images

Thick Cloud Removal of Remote Sensing Images Using Temporal Smoothness and Sparsity Regularized Tensor Optimization

FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors

Contact Info

Product

Resources

About