Efficient Video Compressed Sensing Reconstruction via Exploiting Spatial-Temporal Correlation With Measurement Constraint

Abstract

Recent deep learning-based video compressed sensing (VCS) methods have achieved promising results but still suffer from numerous hyper-parameters and inflexibility. This paper proposes a novel network for VCS, named STM-Net, to fast recover high-quality video frames by optionally exploiting Spatial-Temporal information with a Measurement constraint. Combining the merits of adaptive sampling and adaptive shrinkage-thresholding, we first propose an improved ISTA-Net+ for framewise independent reconstruction, called Unfolding Adaptive Shrinkage-Thresholding Network (UAST-Net). To get further non-key frames reconstruction improvement, we develop a two-phase joint deep reconstruction, including an Occlusion-Aware Temporal Alignment to avoid irrelevant information compensation and a Multiple Frames Fusion with proposed Spatial-Temporal Feature Weighting (STFW) module to guide attractive content extraction and discriminative features generation. Besides, we develop a measurement loss to reduce the solution space to facilitate network optimization. Experimental results demonstrate the superiority of the proposed STM-Net over the existing methods.

A temporal shift reconstruction network for compressive video sensing

“…After the enhancement of initial reconstructions, the alignment is performed on them to build accurate temporal dependencies. Previous methods [13–16, 24, 25, 33] tend to divide the video sequence into several group of pictures (GOP) and only extract supplementary information from the key frames. This framework ignores the correlations between non‐key frames.…”

“…sparse) were imposed on the solution space, which might not exactly represent signal structures. To address these issues, deep learning‐based approaches have recently been introduced in the image/video reconstruction [17–25, 35–38]. Kulkarni et al .…”

“…To solve this problem, Wei et al . [24] employed optical flows gated by occlusion masks to align adjacent frames. Ling et al .…”

“…Since only intra‐frame correlations are considered, the initial reconstruction results in Refs. [24, 25] were too coarse to afford accurate alignments, thus adversely affecting the following fusion process. To address this problem, one intuitive method is refining alignment results and reconstruction results iteratively, but existing alignment manners are too computationally expensive to be repeated too many times.…”

An Overview on Deep Learning Techniques for Video Compressive Sensing

“…In the later work of Weil et al [63], an improved version of ISTA-Net+ is proposed which learns an adaptive sampling matrix by simultaneously optimizing the sampling and reconstruction procedures. A two-phase joint deep reconstruction is adopted to selectively exploit spatial-temporal information, consisting of a temporal alignment with a learnable occlusion mask and a multiple frames fusion with spatial temporal feature weighting (see Figure 8).…”