Estimating the Resize Parameter in End-to-end Learned Image Compression

Abstract

We describe a search-free resizing framework that can further improve the rate-distortion tradeoff of recent learned image compression models. Our approach is simple: compose a pair of differentiable downsampling/upsampling layers that sandwich a neural compression model. To determine resize factors for different inputs, we utilize another neural network jointly trained with the compression model, with the end goal of minimizing the rate-distortion objective. Our results suggest that "compression friendly" downsampled representations can be quickly determined during encoding by using an auxiliary network and differentiable image warping. By conducting extensive experimental tests on existing deep image compression models, we show results that our new resizing parameter estimation framework can provide Bjøntegaard-Delta rate (BD-rate) improvement of about 10% against leading perceptual quality engines. We also carried out a subjective quality study, the results of which show that our new approach yields favorable compressed images. To facilitate reproducible research in this direction, the implementation used in this paper is being made freely available online at: https://github.com/treammm/ResizeCompression.

MaxViT: Multi-axis Vision Transformer

“…These simple approaches, however, can suffer from detail loss and artifacts, and the degraded image quality might hamper the performance of downstream visual recognition tasks, especially when the resizing factor is large. Some recent works have explored to use learning-based methods for image downscaling to enhance the desired content in the resized images from training data [3,4,20,33,45,61]. For example, the authors of [3,4] proposed a residual CNN module for downscaling, and jointly trained it with an image compression network to generate "compressionfriendly" representations.…”

“…2), and discuss how we employ it for training several popular vision tasks. Unlike previous proposed resizers [3,45], we aim to keep the computational cost of the model as low as possible such that it can replace existing resizers (e.g., bilinear) without extra cost, but also there is a notable performance gain. Our proposed approach is different in that (1) it is orders of magnitude faster, hence more scalable (to large image size), (2) it only has a handful of parameters which allows for better generalization, (3) it adds almost no extra training cost to the system.…”

“…To learn more powerful resizing, learned resizers have been proposed [3,4,45] by applying a base resizer on intermediate neural activations, wherein F 1 and F 2 are two designed CNNs applied at the original and output resolutions, respectively. Despite showing promising performance, however, these resizers typically suffer from high computational complexity, and thus their net performance gain might be compromised in terms of the overall inference cost.…”

“…While these simple methods offer greater simplicity and efficiency, they are not optimized for specific computer vision tasks and may lead to the loss of important visual features or details, which can, sometimes, result in significant performance degradation [30,45]. To overcome this limitation, researchers have proposed learned resizers (or downsamplers) [3,45] that leverage the deep neural networks to learn image resizing directly from data, yielding improved performance on several tasks. However, one of the main challenges with these learned resizers is that they often require a large number of parameters, and high computational overhead during training and inference.…”

Blind Video Quality Assessment via Space-Time Slice Statistics

“…The authors of [8,18] studied the temporally bandpass filtered statistics of UGC and variable frame-rate videos. Deep learning has significantly advanced the state-of-the-arts on various vision applications [19][20][21][22]. Recently, several deep learning-based BVQA models have emerged [9,[23][24][25].…”