Two-Stage is Enough: A Concise Deep Unfolding Reconstruction Network for Flexible Video Compressive Sensing

Abstract

We consider the reconstruction problem of video compressive sensing (VCS) under the deep unfolding/rolling structure. Yet, we aim to build a flexible and concise model using minimum stages. Different from existing deep unfolding networks used for inverse problems, where more stages are used for higher performance but without flexibility to different masks and scales, hereby we show that a 2-stage deep unfolding network can lead to the state-of-theart (SOTA) results (with a 1.7dB gain in PSNR over the single stage model, RevSCI [3]) in VCS. The proposed method possesses the properties of adaptation to new masks and ready to scale to large data without any additional training thanks to the advantages of deep unfolding. Furthermore, we extend the proposed model for color VCS to perform joint reconstruction and demosaicing. Experimental results demonstrate that our 2-stage model has also achieved SOTA on color VCS reconstruction, leading to a >2.3dB gain in PSNR over the previous SOTA algorithm based on plugand-play framework, meanwhile speeds up the reconstruction by >17 times. In addition, we have found that our network is also flexible to the mask modulation and scale size for color VCS reconstruction so that a single trained network can be applied to different hardware systems. The code and models will be released to the public.

An Overview on Deep Learning Techniques for Video Compressive Sensing

“…We present a quantitative comparison to compare the quality performances of the following VCS algorithms: GAP-TV [44], DeSCI [73], PnP-FFDNet [74], Pnp-FastDVDNet [88], GAP-FastDVDNet(online) [85], DE-RNN [86], DE-GAP-FFDnet [86], E2E-CNN [48], BIR-NAT [75], MetaSCI [83], RevSCI [82], DeepUnfold-VCS [51], GAP-Unet-S12 [76], ELP-Unfolding [84].…”

“…The idea of measurement calibration algorithm in this approach has improved its final performances on both simulation experiments and optical ones. Another recent work is proposed by Zheng et al [51]. It consists of an encoder-decoder flexible and concise architecture to reconstruct video frames in a CS framework.…”

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

“…In addition to spectral SCI reconstruction as shown in this work, we do believe our network can be used in medical images [ 59 ], image compression [ 60 ], temporal compressive coherent diffraction imaging [ 61 ], and video compressive sensing [ 62 , 63 , 64 , 65 , 66 ].…”

Compressive Video via IR-Pulsed Illumination

“…3. The proposed model is a modified version of the spatio-temporal transformer (STT) neural network, recently proposed in [30], where the initialization module formerly presented in [30][31][32] was replaced with the module shown in Fig. 3(a).…”