DVC: An End-To-End Deep Video Compression Framework

Abstract

Conventional video compression approaches use the predictive coding architecture and encode the corresponding motion information and residual information. In this paper, taking advantage of both classical architecture in the conventional video compression method and the powerful nonlinear representation ability of neural networks, we propose the first end-to-end video compression deep model that jointly optimizes all the components for video compression. Specifically, learning based optical flow estimation is utilized to obtain the motion information and reconstruct the current frames. Then we employ two auto-encoder style neural networks to compress the corresponding motion and residual information. All the modules are jointly learned through a single loss function, in which they collaborate with each other by considering the trade-off between reducing the number of compression bits and improving quality of the decoded video. Experimental results show that the proposed approach can outperform the widely used video coding standard H.264 in terms of PSNR and be even on par with the latest standard H.265 in terms of MS-SSIM.

Video Compression With Rate-Distortion Autoencoders

“…When comparing neural networks to traditional codecs, it is common practice to evaluate those codecs under restrictive settings. For example, group of pictures (GoP) is often set to a value that is similar to the number of frames used to evaluate the neural networks [40,27]. Furthermore, encoding preset will be set to fast (which will result in worse compression performance) [40,27].…”

“…Very recently the video compression problem was attacked by considering flow compression and residual compression [27,33]. The additional components for flow and residual modeling allow to improve distortion in general, however, for low bit rates the proposed method is still outperformed by HEVC/H.265 on benchmark datasets.…”

“…We benchmark our method against the state-of-the-art traditional and learned compression methods on UVG standard test sequences. We compare against classical codecs AVC/H.264 and HEVC/H.265, as well as the recent learned compression methods presented by [27] and [40]. For the classical codecs, we use the default FFmpeg settings, without imposing any restriction, and only vary the CRF setting to obtain rate/distortion curves.…”

“…For our neural network, we use the rate loss (converted into bpp) as a proxy for rate. By definition, the rate loss gives the expected bitrate under adaptive arithmetic coding, and expected bpp was shown to be highly correlated to actual bpp [27].…”

Feedback Recurrent Autoencoder

“…It is worth noting that the video compression framework of DVC [13] can be viewed as an instantiation of Fig. 1(f) where decoded data in the previous time steps are fed back to the encoder for explicit motion and residual information compression, and the one proposed in VQ-VAE [7,8] can be viewed as a convolutional variant of Fig.…”