Compressed Video Sensing Network Based On Alignment Prediction And Residual Reconstruction

Ling, Xi; Yang, Chunling; Pei, Hanqi

doi:10.1109/icme46284.2020.9102723

Cited by 6 publications

(12 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the fairness of experimental comparison, as in [12,13,14,15], we use the UCF-101 dataset for model training and randomly divide all video sequences into 80% as the training set, 10% as the validation set and 10% as the test set. Each frame is centrally cropped to a size of 160 × 160 and only the lumimance component in the yCbCr color space is retained.…”

Section: Resultsmentioning

confidence: 99%

“…The results of other neural network methods are obtained from the corresponding published papers. In [12,15,14], GOP size (T) is 10, the sampling rate of key frames is 0.2, and the sampling rate of non-key frames is 0.037, 0.018 and 0.009. In [13], the GOP size (T) is 4, the sampling rate of key frames is 0.25, and the sampling rate of non-key frames is 0.1, 0.04, and 0.01.…”

Section: Comparison With Deep Learning Methodsmentioning

confidence: 99%

“…Neural network can be used to learn data-driven transformation coefficients to achieve high sparsity transform. Current deep learning methods have achieved better performance than traditional methods in the field of image compressed sensing [9,10,11] and video compressed sensing [12,13,14,15] while being significantly faster at reconstruction time. Through stacking some naive inception modules and convolutional layers, JsrNet [13] directly learns the nonlinear mapping of image groups from the observation domain to pixel domain.…”

Section: Introductionmentioning

confidence: 99%

“…The above two end-to-end methods lack a specialized pixel alignment network so that the quality of motion compensation for non-key frame is poor. In order to make full use of the pixel domain information of the key frame, Ling et al applied Temporally Deformable Alignment Network (TDAN) [16] as motion compensation to video compressed sensing and constructed multiple residual reconstruction blocks to learn the nonlinear mapping from sampling error to real error [14]. Xuan et al used the Spatial Transformer Motion Compensation (STMC) [17] to perform motion compensation and constructed a sparse representation of video inter-frame groups by a neural network, called VGSR-Net, which is used for image reconstruction and residual reconstruction [15].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Imrnet: An Iterative Motion Compensation and Residual Reconstruction Network for Video Compressed Sensing

Yang

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

The traditional video compressed sensing (VCS) algorithms have elegant theoretical interpretability. However, the deterministic sparse transformation used in these algorithms usually can not satisfy the sparsity need, which results in poor reconstruction quality. Also, the optimization process is slow. Deep learning can learn data-driven transformation while achieving fast reconstruction. This paper proposes an iterative motion compensation and residual reconstruction network for VCS, called ImrNet. ImrNet follows the iterative optimization method of MC-BCS-SPL framework, and each module in ImrNet is trained independently. In addition, we design a motion-compensated network (MCNet) to achieving adaptive fusion among frames by using the image semantic segmentation method to obtain the probability map as fusion weights of each frame. The proposed MCNet is used to generate fusion-compensated frame in each iteration of ImrNet. Experimental results show that ImrNet can achieve good reconstruction results with only two iterations. Its reconstruction quality is better than state-of-the-art VCS methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Comparison With Deep Learning Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Imrnet: An Iterative Motion Compensation and Residual Reconstruction Network for Video Compressed Sensing

Yang

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Motivated by the success of convolutional neural network (CNN) in image CS (ICS), some CNN-based VCS methods [7,8,9,10,11] have been proposed recently and achieved superior performance to traditional methods. In [7], Xu et al proposed CSVideoNet that adopts a long short-term memory (LSTM) network to extract motion features among adjacent frames for VCS reconstruction.…”

Section: Introductionmentioning

confidence: 99%

Efficient Video Compressed Sensing Reconstruction via Exploiting Spatial-Temporal Correlation With Measurement Constraint

Zhou

Yang

Xuan

2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

Self Cite

View full text Add to dashboard Cite

Recent deep learning-based video compressed sensing (VCS) methods have achieved promising results but still suffer from numerous hyper-parameters and inflexibility. This paper proposes a novel network for VCS, named STM-Net, to fast recover high-quality video frames by optionally exploiting Spatial-Temporal information with a Measurement constraint. Combining the merits of adaptive sampling and adaptive shrinkage-thresholding, we first propose an improved ISTA-Net+ for framewise independent reconstruction, called Unfolding Adaptive Shrinkage-Thresholding Network (UAST-Net). To get further non-key frames reconstruction improvement, we develop a two-phase joint deep reconstruction, including an Occlusion-Aware Temporal Alignment to avoid irrelevant information compensation and a Multiple Frames Fusion with proposed Spatial-Temporal Feature Weighting (STFW) module to guide attractive content extraction and discriminative features generation. Besides, we develop a measurement loss to reduce the solution space to facilitate network optimization. Experimental results demonstrate the superiority of the proposed STM-Net over the existing methods.

show abstract

A temporal shift reconstruction network for compressive video sensing

Gu,

Zhou,

Lin

2023

IET Computer Vision

View full text Add to dashboard Cite

Compressive sensing provides a promising sampling paradigm for video acquisition for resource‐limited sensor applications. However, the reconstruction of original video signals from sub‐sampled measurements is still a great challenge. To exploit the temporal redundancies within videos during the recovery, previous works tend to perform alignment on initial reconstructions, which are too coarse to provide accurate motion estimations. To solve this problem, the authors propose a novel reconstruction network, named TSRN, for compressive video sensing. Specifically, the authors utilise a number of stacked temporal shift reconstruction blocks (TSRBs) to enhance the initial reconstruction progressively. Each TSRB could learn the temporal structures by exchanging information with last and next time step, and no additional computations is imposed on the network compared to regular 2D convolutions due to the high efficiency of temporal shift operations. After the enhancement, a bidirectional alignment module to build accurate temporal dependencies directly with the help of optical flows is employed. Different from previous methods that only extract supplementary information from the key frames, the proposed alignment module can receive temporal information from the whole video sequence via bidirectional propagations, thus yielding better performance. Experimental results verify the superiority of the proposed method over other state‐of‐the‐art approaches quantitatively and qualitatively.

show abstract

Compressed Video Sensing Network Based On Alignment Prediction And Residual Reconstruction

Cited by 6 publications

References 12 publications

Imrnet: An Iterative Motion Compensation and Residual Reconstruction Network for Video Compressed Sensing

Imrnet: An Iterative Motion Compensation and Residual Reconstruction Network for Video Compressed Sensing

Efficient Video Compressed Sensing Reconstruction via Exploiting Spatial-Temporal Correlation With Measurement Constraint

A temporal shift reconstruction network for compressive video sensing

Contact Info

Product

Resources

About