Multi-frame Quality Enhancement for Compressed Video

Abstract

The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, ignoring the similarity between consecutive frames. In this paper, we investigate that heavy quality fluctuation exists across compressed video frames, and thus low quality frames can be enhanced using the neighboring high quality frames, seen as Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as a first attempt in this direction. In our approach, we firstly develop a Support Vector Machine (SVM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are as the input. The MF-CNN compensates motion between the non-PQF and PQFs through the Motion Compensation subnet (MC-subnet). Subsequently, the Quality Enhancement subnet (QE-subnet) reduces compression artifacts of the non-PQF with the help of its nearest PQFs. Finally, the experiments validate the effectiveness and generality of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video. The code of our MFQE approach is available at

Non-Local ConvLSTM for Video Compression Artifact Reduction

“…To evaluate our method, we conduct extensive experiments on two datasets: Vimeo-90K [42] and Yang et al's dataset [45]. Our evaluation consists of five parts: 1) Ablation study; 2) Quantitative evaluation with two performance metrics (PSNR and SSIM) ; 3) Qualitative evaluation by comparing the visual effect of compression artifact reduction of different methods; 4) Run time comparison; 5) Checking the effectiveness of our method on videos compressed by another algorithm.…”

“…Resolutions of these video sequences vary from 352 × 240 to 2, 560 × 1, 600. For a fair comparison, we follow the settings in [45]: 60 sequences are taken for training and the remaining 10 for testing. All sequences are encoded in HEVC LDP mode, using HM 16.0 with QP =37 and 42.…”

“…For DKFN [26], we directly cite performance results from the original paper where it was evaluated only on Vimeo-90K. For MFQE [45], we cite the results of PSNR improvement, and compute SSIM improvement via our manually labeled PQFs and its published model 3 . Besides, considering that 3D CNN is able to capture spatiotemporal information from video frames, we also implement it for performance comparison by adopting the same architecture as [17] that aims at video super-resolution.…”

“…From Table 3, we can see that our method outperforms the 2 nd best methods (DKFN and 3D CNN) by about 0.14 dB in terms of PSNR and 25% improvement on ∆SSIM. These methods that utilize spatiotemporal information, including ours, DKFN and 3D CNN, all achieve better performance 3 [45] only publishes the model with QP =37. Table 4 we can still get similar conclusion: methods exploiting spatiotemporal information of neighboring frames perform better than these do not exploit.…”

“…Also, comparing the details in the boxes of the same color, we can see that the cropped patches from the 142 th frame are of better quality. emerged as an important research topic in multimedia and computer vision areas [26,43,45].…”