CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement

Ma, Di; Zhang, Fan; Bull, David

doi:10.48550/arxiv.2011.09190

Cited by 7 publications

(7 citation statements)

References 91 publications

(178 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to capture pixel movements at multiple scales, we devise a U-Net style feature extractor, U-MultiScaleResNext (UMSResNext), consisting of eight MSResNext blocks (illustrated in Figure 3). Each MSRes-Next block employs two ResNext blocks [59] in parallel with different kernel sizes in the middle layer, 3×3 and 7×7, which further increases the network cardinality [37,59]. The outputs of these two ResNext blocks are then concatenated and connected to a channel attention module [24], which learns adaptive weighting of the feature maps extracted by the two ResNext blocks.…”

Section: Multi-interflow Networkmentioning

confidence: 99%

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Danier¹,

Zhang²,

Bull³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Video frame interpolation (VFI) is currently a very active research topic, with applications spanning computer vision, post production and video encoding. VFI can be extremely challenging, particularly in sequences containing large motions, occlusions or dynamic textures, where existing approaches fail to offer perceptually robust interpolation performance. In this context, we present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture. ST-MFNet employs a new multi-scale multi-flow predictor to estimate many-to-one intermediate flows, which are combined with conventional one-to-one optical flows to capture both large and complex motions. In order to enhance interpolation performance for various textures, a 3D CNN is also employed to model the content dynamics over an extended temporal window. Moreover, ST-MFNet has been trained within an ST-GAN framework, which was originally developed for texture synthesis, with the aim of further improving perceptual interpolation quality. Our approach has been comprehensively evaluated -compared with fourteen state-of-the-art VFI algorithms -clearly demonstrating that ST-MFNet consistently outperforms these benchmarks on varied and representative test datasets, with significant gains up to 1.09dB in PSNR for cases including large motions and dynamic textures. Project page: https://danielism97.github.io/ST-MFNet.

show abstract

Section: Multi-interflow Networkmentioning

confidence: 99%

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Danier¹,

Zhang²,

Bull³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The spatio-temporal filter adaptive network was developed for video deblurring [45] and optical flow estimation was jointly trained with processing component for video enhancement [41]. Recently, generative adversarial network (GAN) also attracted considerable attention due to its favorable performance especially for extremely low bit-rate compression [5,23].…”

Section: Related Workmentioning

confidence: 99%

Multi-Density Attention Network for Loop Filtering in Video Compression

Wang,

Ma,

2021

Preprint

View full text Add to dashboard Cite

Video compression is a basic requirement for consumer and professional video applications alike. Video coding standards such as H.264/AVC and H.265/HEVC are widely deployed in the market to enable efficient use of bandwidth and storage for many video applications. To reduce the coding artifacts and improve the compression efficiency, neural network based loop filtering of the reconstructed video has been developed in the literature. However, loop filtering is a challenging task due to the variation in video content and sampling densities. In this paper, we propose a on-line scaling based multi-density attention network for loop filtering in video compression. The core of our approach lies in several aspects: (a) parallel multi-resolution convolution streams for extracting multi-density features, (b) single attention branch to learn the sample correlations and generate mask maps, (c) a channel-mutual attention procedure to fuse the data from multiple branches, (d) on-line scaling technique to further optimize the output results of network according to the actual signal. The proposed multi-density attention network learns rich features from multiple sampling densities and performs robustly on video content of different resolutions. Moreover, the online scaling process enhances the signal adaptability of the off-line pre-trained model. Experimental results show that 10.18% bit-rate reduction at the same video quality can be achieved over the latest Versatile Video Coding (VVC) standard. The objective performance of the proposed algorithm outperforms the state-of-the-art methods and the subjective quality improvement is obvious in terms of detail preservation and artifact alleviation. CCS CONCEPTS• Computing methodologies → Image compression.

show abstract

“…Since TMIV is based on a 2D video coding framework, we hypothesise that it would benefit from resolution adaptation methods [9,10,11,12,13] and post-processing methodologies [14,15] that have proved successful in conventional coding. In this context we propose a novel approach that exploits resolution adaptation and post-processing of synthesized views, realised using a Convolutional Neural Network (CNN).…”

Section: Introductionmentioning

confidence: 99%

A CNN-based Post-Processor for Perceptually-Optimized Immersive Media Compression

Katsenou¹,

Zhang²,

Bull³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, resolution adaptation based on deep neural networks has enabled significant performance gains for conventional (2D) video codecs. This paper investigates the effectiveness of spatial resolution resampling in the context of immersive content. The proposed approach reduces the spatial resolution of input multi-view videos before encoding, and reconstructs their original resolution after decoding. During the up-sampling process, an advanced CNN model is used to reduce potential re-sampling, compression and synthesis artefacts. This work has been fully tested with the TMIV coding standard using a Versatile Video Coding (VVC) codec. The results demonstrate that the proposed method achieves significant rate quality performance improvement for the majority of the test sequences, with an average BD-VMAF improvement of 3.07 over all sequences.

show abstract

CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement

Cited by 7 publications

References 91 publications

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Multi-Density Attention Network for Loop Filtering in Video Compression

A CNN-based Post-Processor for Perceptually-Optimized Immersive Media Compression

Contact Info

Product

Resources

About