FVC: A New Framework towards Deep Video Compression in Feature Space

Abstract

Learning based video compression attracts increasing attention in the past few years. The previous hybrid coding approaches rely on pixel space operations to reduce spatial and temporal redundancy, which may suffer from inaccurate motion estimation or less effective motion compensation. In this work, we propose a feature-space video coding network (FVC) by performing all major operations (i.e., motion estimation, motion compression, motion compensation and residual compression) in the feature space. Specifically, in the proposed deformable compensation module, we first apply motion estimation in the feature space to produce motion information (i.e., the offset maps), which will be compressed by using the auto-encoder style network. Then we perform motion compensation by using deformable convolution and generate the predicted feature. After that, we compress the residual feature between the feature from the current frame and the predicted feature from our deformable compensation module. For better frame reconstruction, the reference features from multiple previous reconstructed frames are also fused by using the nonlocal attention mechanism in the multi-frame feature fusion module. Comprehensive experimental results demonstrate that the proposed framework achieves the state-of-the-art performance on four benchmark datasets including HEVC, UVG, VTL and MCL-JCV.

A Coding Framework and Benchmark towards Compressed Video Understanding

“…For the traditional video codecs [52] [53], different linear transformations are exploited to better capture the statistical characteristics of the texture and motion information within the videos. Latter, learnable video codecs [54] [55] [56] [57] [58] [59] [60] gain increasing attention. Following the traditional hybrid video compression framework, Lu et al [54] proposed the first endto-end optimized video compression framework, in which all the key components in H.264/H.265 are replaced with deep neural networks.…”

Implicit Neural Video Compression

“…[24,49], which use 3D convolution architectures, and Refs. [3,9,11,12,16,23,26,33,36,37,48,50,52,53,69], which model P-frames as an optical flow field applied to the previous frame plus a residual model.…”

“…first estimate of the current frame. In addition to the optical flow, pixel-level [3,37] or feature-level [26] residuals are compressed. The final prediction for the frame is given by applying the optical flow field to the previous reconstructed frame in an operation known as warping or motion compensation and adding the residuals.…”

AIVC: Artificial Intelligence based Video Codec

“…Most previous works focus on the low-delay P configuration [6,7] (used for videoconferencing) and omit the Random Access configuration (used for streaming at large). Furthermore, most neural codecs [10,11] are assessed using an I frame period shorter (e.g. 10 or 12 frames, regardless of the video framerate) than expected by the Common Test Conditions of modern video coders such as those defined for HEVC or VVC [12].…”