Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

Abstract

In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motions of multiple frames, thus saving bits for motion information. In our deep decoder, we develop the Weighted Recurrent Quality Enhancement (WRQE) network, which takes both compressed frames and the bit stream as inputs. In the recurrent cell of WRQE, the memory and update signal are weighted by quality features to reasonably leverage multiframe information for enhancement. In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides, respectively. Finally, the experiments validate that our HLVC approach advances the stateof-the-art of deep video compression methods, and outperforms the "Low-Delay P (LDP) very fast" mode of x265 in terms of both PSNR and MS-SSIM. The project page is at https://github.com/RenYang-home/HLVC.

Deep Predictive Video Compression Using Mode-Selective Uni- and Bi-Directional Predictions Based on Multi-Frame Hypothesis

“…The DeepPVCnet is compared with the conventional video codecs such as AVC/H.264 and HEVC/H.265, as well as three deep learning-based video compression methods in [4], [13], [25], [26], [46], [47]. For fair comparison, the GOP size of the conventional video codecs is fixed to 12.…”

“…It shows good performance in high bit ranges, but not in low or mid bit ranges. In [47], Yang et al proposed a hierarchical learned video compression method with the hierarchical quality layers and a recurrent enhancement network. The method in [13] incorporates a 3D auto-encoder that does not use the P-frame and B-frame coding concept.…”

“…In addition, we used a batch size of 8 and a patch of 256×256 randomly cropped from the 466K training frames extracted from the UGC video dataset. [38], and CNN-based SOTA methods [4], [13], [25], [26], [46], [47] for the UVG dataset, HEVC Class B and E dataset. Evaluation.…”

“…In [10], Cui et al proposed intra-prediction method with CNN in HEVC to improve compression performance. In [51], Zhao et al replaced the bi-prediction strategy in HEVC with CNN to improve coding efficiency; Second, there are studies to improve the compression performance by using auto-encoder based end-to-end neural network architectures [4], [9], [11], [13], [25], [26], [36], [46], [47]. Although deep learningbased image compression has been intensively studied, video compression has drawn less attention.…”

“…• Contrary to the deep neural network-based state-of-theart (SOTA) methods [26], [46] that reply on a single reference frame for each prediction direction, our method improves prediction accuracy for the current frame by utilizing multiple frames for both uni-and bi-directional prediction modes. This paper is organized as follows: Section II introduces the related work with deep neural network-based image/video compression, optical flow estimation and frame interpolation; In Section III, we introduce the details of our proposed deep video compression network, called DeepPVCnet; Section IV presents the experimental results to show the effectiveness of our DeepPVCnet compared to the conventional video codecs and SOTA methods [4], [13], [25], [26], [46], [47]; Finally, we conclude our work in Section V.…”