A DenseNet Based Approach for Multi-frame In-loop Filter in HEVC

Li, Tianyi; Xu, Mai; Yang, Ren; Tao, Xiaoming

doi:10.1109/dcc.2019.00035

“…Moreover, the proposed method can inspire traditional codecs, particularly the methods that integrate deep networks in traditional codecs, to adopt recurrent networks to improve their performance. For instance, Liu et al [41] and Choi et al [42] improves the motion compensation of HEVC by utilizing single deep network on each frame, and Li et al [44], [45] replace the in-loop of HEVC by non-recurrent deep networks. These methods are possible to be advanced by employing the recurrent networks (similar to the proposed approach) to further improve the traditional codecs, such as HEVC.…”

Section: Discussionmentioning

confidence: 99%

Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model

Yang

¹

,

Mentzer

²

,

Gool

³

et al. 2021

IEEE J. Sel. Top. Signal Process.

Self Cite

View full text Add to dashboard Cite

The past few years have witnessed increasing interests in applying deep learning to video compression. However, the existing approaches compress a video frame with only a few number of reference frames, which limits their ability to fully exploit the temporal correlation among video frames. To overcome this shortcoming, this paper proposes a Recurrent Learned Video Compression (RLVC) approach with the Recurrent Auto-Encoder (RAE) and Recurrent Probability Model (RPM). Specifically, the RAE employs recurrent cells in both the encoder and decoder. As such, the temporal information in a large range of frames can be used for generating latent representations and reconstructing compressed outputs. Furthermore, the proposed RPM network recurrently estimates the Probability Mass Function (PMF) of the latent representation, conditioned on the distribution of previous latent representations. Due to the correlation among consecutive frames, the conditional cross entropy can be lower than the independent cross entropy, thus reducing the bit-rate. The experiments show that our approach achieves the state-of-the-art learned video compression performance in terms of both PSNR and MS-SSIM. Moreover, our approach outperforms the default Low-Delay P (LDP) setting of x265 on PSNR, and also has better performance on MS-SSIM than the SSIM-tuned x265 and the slowest setting of x265.The code and pre-trained models will be released on the project page: https://github.com/RenYang-home/RLVC.git.

show abstract

“…Many approaches [40,21,11,19,20] were proposed to replace the components in traditional video codecs by DNNs. For instance, Liu et al [21] utilized a DNN in the fractional interpolation of motion compensation, and [11,19,20] use DNNs to improve the in-loop filter. However, these methods only advance the performance of one particular module, and each module in video compression framework cannot be jointly optimized.…”

Section: Deep Video Compressionmentioning

confidence: 99%

Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

Yang

¹

,

Mentzer

²

,

Gool

³

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motions of multiple frames, thus saving bits for motion information. In our deep decoder, we develop the Weighted Recurrent Quality Enhancement (WRQE) network, which takes both compressed frames and the bit stream as inputs. In the recurrent cell of WRQE, the memory and update signal are weighted by quality features to reasonably leverage multiframe information for enhancement. In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides, respectively. Finally, the experiments validate that our HLVC approach advances the stateof-the-art of deep video compression methods, and outperforms the "Low-Delay P (LDP) very fast" mode of x265 in terms of both PSNR and MS-SSIM. The project page is at https://github.com/RenYang-home/HLVC.

show abstract

“…In recent years, deep learning also attracted more attention in video compression. Many approaches [40,21,11,19,20] were proposed to replace the components in traditional video codecs by DNNs. For instance, Liu et al [21] utilized a DNN in the fractional interpolation of motion compensation, and [11,19,20] use DNNs to improve the in-loop filter.…”

Section: Deep Video Compressionmentioning

confidence: 99%

Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement

Yang

¹

,

Mentzer

²

,

Gool

³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motions of multiple frames, thus saving bits for motion information. In our deep decoder, we develop the Weighted Recurrent Quality Enhancement (WRQE) network, which takes both compressed frames and the bit stream as inputs. In the recurrent cell of WRQE, the memory and update signal are weighted by quality features to reasonably leverage multiframe information for enhancement. In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides, respectively. Finally, the experiments validate that our HLVC approach advances the stateof-the-art of deep video compression methods, and outperforms the "Low-Delay P (LDP) very fast" mode of x265 in terms of both PSNR and MS-SSIM. The project page is at https://github.com/RenYang-home/HLVC.

show abstract

A DenseNet Based Approach for Multi-frame In-loop Filter in HEVC

Cited by 17 publications

References 16 publications

Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model

Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model

Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement

Contact Info

Product

Resources

About