Deep Learning Post-Filtering Using Multi-Head Attention and Multiresolution Feature Fusion for Image and Intra-Video Quality Enhancement

Abstract

The paper proposes a novel post-filtering method based on convolutional neural networks (CNNs) for quality enhancement of RGB/grayscale images and video sequences. The lossy images are encoded using common image codecs, such as JPEG and JPEG2000. The video sequences are encoded using previous and ongoing video coding standards, high-efficiency video coding (HEVC) and versatile video coding (VVC), respectively. A novel deep neural network architecture is proposed to estimate fine refinement details for full-, half-, and quarter-patch resolutions. The proposed architecture is built using a set of efficient processing blocks designed based on the following concepts: (i) the multi-head attention mechanism for refining the feature maps, (ii) the weight sharing concept for reducing the network complexity, and (iii) novel block designs of layer structures for multiresolution feature fusion. The proposed method provides substantial performance improvements compared with both common image codecs and video coding standards. Experimental results on high-resolution images and standard video sequences show that the proposed post-filtering method provides average BD-rate savings of 31.44% over JPEG and 54.61% over HEVC (x265) for RGB images, Y-BD-rate savings of 26.21% over JPEG and 15.28% over VVC (VTM) for grayscale images, and 15.47% over HEVC and 14.66% over VVC for video sequences.

Improving Compressed Video Using Single Lightweight Model with Temporal Fusion Module

“…The architecture of deep learning-based restoration methods are commonly made up of three layers: feature extraction, feature processing, and reconstruction. To realize the reconstruction layer, two approaches based on the output of the predicted content can be used: by predicting the whole restored image [6,[9][10][11]13,14] or by predicting the residual information for compensating the artifacts [5,7,12,[15][16][17]19,20]. Because the amount of information required to predict residual information is less, the residual information prediction is less difficult and the accuracy can be improved.…”

“…It proves the importance of restoring color artifacts, which is why we developed our method for processing both the luminance and chroma. Table 4 compares the results of our method to those of other studies [10,[12][13][14][15][16][17] tested in the LIVE1 dataset. Since no other existing work provides data for the chroma channel, we will only compare our chroma improvement to the chroma quality of JPEG-compressed images.…”

“…Several researchers [10][11][12] designed shallow models to retrieve spatial information and generate restoration results. Unfortunately, the restoration results were unsatisfactory, and thus, a deeper network architecture was adopted [13][14][15][16][17]. Cavigelli et al [13] adjusted the filter stride in the network to process the feature maps at four different scales to accommodate different object sizes.…”