FM-VSR: Feature Multiplexing Video Super-Resolution for Compressed Video

Abstract

Due to the limitation of shooting conditions in the real world, there exist many insufficientresolution videos. To be transmitted under limited bandwidth conditions, low-resolution videos often have to be compressed further, which introduces more compression artifacts and results in severer damage to the video quality. Accordingly, video super-resolution in practical applications must repair compressed damage and down-sampling damage simultaneously, which can be intuitively solved by two subtasks, including compressed video quality enhancement (VQE) and video super-resolution (VSR). Therefore, we proposed a novel model, the Feature Multiplexing Video Super-Resolution for Compressed Video (FM-VSR) to effectively handle such multi-tasks. Firstly, we use a complicated Deformable Regression Pyramid (DRP) module to align the reference frame and each supporting frame. Then feature multiplexing structure is utilized in VSR to make better use of the VQE information. Finally, a skip-attention structure is introduced in the reconstruction module to predict the high-quality video frame. Many experiments on benchmark datasets show that our method can achieve better performance both in quantitative and qualitative than the state-of-the-art (SOTA) two-stage approaches. Super-resolution, video compression, deep learning. INDEX TERMS

Edge-Oriented Compressed Video Super-Resolution

“…The performance of our proposed EOCVSR is compared with the latest CVSR approaches, including RR-DnCNN v2 [ 14 ] and FM-VSR [ 13 ]. As shown in Table 2 , we compare the characteristics between EOCVSR and other CVSR approaches.…”

“…First, the results illustrate that the proposed EOCVSR outperforms all the prior approaches over four compression ratios. Specifically, the PSNR of EOCVSR is 29.037 dB, which is 0.038 dB higher than the state-of-the-art approach, i.e., BasicVSR++ (28.999 dB), and 0.107-0.325 dB higher than others [8,13,14]. Table 5 shows the PSNR results over the UVG and MCL-JCV datasets, and the results demonstrate that EOCVSR also achieves the most significant boost.…”

“…This architectural modification enables the propagation of valuable features captured during the restoration process to the subsequent reconstruction stage. He et al proposed a one-stage video super-resolution network (FM-VSR) [ 13 ] to recover high-quality, high-resolution videos from low-resolution videos. They make full use of the internal correlation between the VQE and VSR tasks by adopting a feature multiplexing mechanism.…”

“…The main reason is that the information of the two independent processes cannot be effectively collaborated. Recently, there has been a growing interest in the task of compressed video super-resolution (CVSR) [ 12 , 13 , 14 ], which sought to address both compression artifacts removal and resolution enhancement in compressed videos simultaneously. An end-to-end restoration-reconstruction deep neural network (RR-DnCNN) [ 12 ] was first proposed to entirely solve degradation from compression and downsampling.…”

“…The authors utilized upsampling skip connections to pass the useful features extracted by restoration to reconstruction. He et al introduced a novel model, named Feature Multiplexing Video Super-Resolution for Compressed Video (FM-VSR) [ 13 ], to recover high-quality, high-resolution videos from low-resolution videos compressed with high compression rates. However, these methods lack targeted processing for high-frequency components, particularly edge components.…”