A CNN-Based Post-Processing Algorithm for Video Coding Efficiency Improvement

Abstract

Lossy compression algorithms are widely used in video coding. However, lossy compressed videos exist some annoying distortion and artifacts, such as blocking, blurring, and ringing. Thus, coding efficiency improvement is a steady-state topic in the domain of video coding. High Efficiency Video Coding (HEVC), a recent video standard, adopts two in-loop filters for the improvement of the coding efficiency, including deblocking (DB) and sample adaptive offset (SAO). In a certain extent, traditional in-loop filters reduce the distortion and improve the video quality. But the reduction of the distortion is a nonlinear problem that is difficult to be solved by traditional linear filters. Recently, the progress of deep learning shows the possibility to settle the complex problems in the computer vision field. Meanwhile, according to the compressive sensing theory, the post-processing method at the decoder end can further enhance the coding efficiency. In this paper, we propose a variable-filter-size Residue-learning convolutional neural network with batch normalization layer (VRCNN-BN). Our model is an end-to-end model. We feed the decoded pictures to the model at the decoder end. Different from previous methods, we apply the model to luma pictures and chroma pictures, respectively. In order to comprehensively evaluate the coding performance of both luma and chroma components, the color-sensitivity-based combined PSNR (CS-PSNR) is exploited to measure the effectiveness of the proposed method. Compared to HEVC baseline, our approach achieves an average BD-rate reduction of 10.3%, 8.9%, 13.1% and 11.8% in terms of CS-PSNR for random access, all intra, low delay P and low delay B configurations, respectively. Abundant experimental results indicate that our method is better than existing similar methods. INDEX TERMS Convolutional neural network, end-to-end, post-processing, high efficiency video coding.

A CNN-Based Prediction-Aware Quality Enhancement Framework for VVC

“…Inspired by the diversity of block sizes in HEVC, an ILF named Variable-filter-size Residue-learning CNN (VRCNN) proposes a network with different filter sizes to replace both SAO and DBF filters of HEVC intra coding [17]. The method presented in [30] enhances the performance of VRCNN by introducing more non-linearity to the VRCNN network. The added ReLU [62] and batch normalization [63] layers in this method improve its performance, compared to VRCNN.…”

Prediction-Aware Quality Enhancement of VVC Using CNN

“…CTCs QP (22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37) High QP ( the effect of other coding information. Therefore, as another future work, one can benefit from other coding information for further improving the proposed QE framework.…”

CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement

“…A second class contains algorithms (primarily based on CNNs) that are designed to enhance individual coding tools within a standard codec configuration. Such approaches have been used to optimise tools including: intra prediction [23,24], motion estimation [25,26], transforms [27,28], quantisation [29], entropy coding [30,31], post-processing (PP) [32,33], in-loop filtering (ILF) [34,35] and format adaptation [36][37][38]. Among these CNN-based coding tools, there is one group of methods, which stand out in offering higher coding gains compared to the others [2,3].…”

“…We have used an eight-fold cross validation method [77] to train the proposed combined loss function (equation ( 1)) based on eight publicly available subjective video quality databases. These include: the Netflix public database (70 test sequences) [78], BVI-HD (192) [67], CC-HD (108) [11], CC-HDDO (90) [79], MCL-V (96) [80], SHVC (32) [81], IVP (100) [82], and VQEG-HD3 (72) [83]. All of these contain video sequences compressed using commonly used video codecs (H.264, HEVC, AV1, VVC or MPEG-2).…”