Video Compression With Rate-Distortion Autoencoders

Abstract

In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a ratedistortion loss, which is closely related to the ELBO used in variational autoencoders. Despite its simplicity, we find that our method outperforms the state-of-the-art learned video compression networks based on motion compensation or interpolation. We systematically evaluate various design choices, such as the use of frame-based or spatio-temporal autoencoders, and the type of autoregressive prior.In addition, we present three extensions of the basic method that demonstrate the benefits over classical approaches to compression. First, we introduce semantic compression, where the model is trained to allocate more bits to objects of interest. Second, we study adaptive compression, where the model is adapted to a domain with limited variability, e.g. videos taken from an autonomous car, to achieve superior compression on that domain. Finally, we introduce multimodal compression, where we demonstrate the effectiveness of our model in joint compression of multiple modalities captured by non-standard imaging sensors, such as quad cameras. We believe that this opens up novel video compression applications, which have not been feasible with classical codecs.

Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

“…Recent studies in learned image compression, e.g., [2,3], show the great potential of deep learning for improving the rate-distortion performance. It is therefore not surprising to see increasing interest in compressing video with Deep Neural Networks (DNNs) [8,38,9,22,13]. For example, Lu et al [22] proposed using optical flow for motion compensation and applying auto-encoders to compress the flow and residual.…”

[Paper] HEVC-based Light-field Coding using Basis Images and Frame Reordering

“…Instead of the current video coding standard (HEVC), the upcoming video coding standard called Versatile Video Coding (VVC) [28] can also be combined with our method. We also intend to investigate other coding methods including 3D HEVC [7,10], and learned video coding methods [19][20][21][22] to clarify the strength of each method and to evaluate the potential synergy with our method. Finally, we hypothesize that generating basis images is also effective for other applications such as coding of high-speed videos.…”

“…The experimental results show that this method achieves better rate-distortion performance than other HEVC-based LF coding methods [12][13][14]18]. Reflecting the recent explosive prevalence of deep neural networks, learning-based video coding methods [19][20][21][22] have attracted much interest. However, these methods are still struggling to outperform the sophisticated video codecs.…”

“…However, these methods are still struggling to outperform the sophisticated video codecs. For example, although the method developed by Habibian et al [22] has performance comparable to that of HEVC, their anchor, referred to as "HEVC", was actually the FFmpeg with the default configuration, which has a performance substantially inferior to that of the well-tuned HEVC reference software, HM [23], maintained by the Joint Collaborative Team on Video Coding (JCT-VC). Meanwhile, our method can incorporate any HEVC codecs.…”

Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video

“…[30], however, we use pointclouds for the front layers and a closed mesh for the backmost layer. Furthermore, instead of directly accessing and optimizing depth information, we adapt ideas from autoencoders [9] to store scene depths in a neural network. Our network compresses the required depth information and offers an efficient way to optimize for temporal and spatial smoothness.…”