Deep Residual Learning in the JPEG Transform Domain

Abstract

We introduce a general method of performing Residual Network inference and learning in the JPEG transform domain that allows the network to consume compressed images as input. Our formulation leverages the linearity of the JPEG transform to redefine convolution and batch normalization with a tune-able numerical approximation for ReLu. The result is mathematically equivalent to the spatial domain network up to the ReLu approximation accuracy. A formulation for image classification and a model conversion algorithm for spatial domain networks are given as examples of the method. We show skipping the costly decompression step allows for faster processing of images with little to no penalty in the network accuracy.1. The general method for expressing convolutional networks in the JPEG domain 2. Concrete formulation for residual blocks to perform classification 3. A model conversion algorithm to apply pretrained spatial domain networks to JPEG images 4. Approximated Spatial Masking: the first general technique for application of piecewise linear functions in the transform domain By skipping the decompression step and by operating on the compressed format, we show a notable increase in speed for testing and a marginal speed for training. Prior WorkWe review prior work separated into three categories: compressed domain operations, machine learning in the compressed domain, and deep learning in the compressed domain. Compressed Domain OperationsThe expression of common operations in the compressed domain was an extremely active area of study in the late 80s and early 90s, motivated by the lack of computing power to 1

Fast object detection in compressed JPEG Images

“…The resulting networks are 1.77× faster at inference and attain state-of-the-art classification performances. Another research stream designs dedicated networks to spectral input coefficients: harmonic networks [6] uses custom convolutions that produce high-level features by learning combinations of spectral filters defined by the 2D Discrete Cosine Transform; Ehrlich and Davis (2019) [7] introduce a ResNet able to operate on compressed JPEG images by including the compression transform into the network weights. From video side, two recent works on detection in compressed videos are [8], [9].…”

MGCAN: A Novel Method for Visual Power Monitoring Systems

“…Generally, when an image is fully focused, the image is clearest and the high-frequency components are maximized. The frequency domain contains rich patterns that are useful for imageunderstanding tasks [28]- [30]. Accordingly, it follows that using frequency-domain information could enhance task performance.…”

Context-Adaptive Inverse Quantization for Inter-Frame Coding

“…Chen et al [24] pointed out that filtering the transform coefficients is a more direct way to compensate for the quantization loss, and it is helpful to consider the consistency with the human visual system. Studies in [25]- [27] show that it is feasible to use deep neural networks to process DCTdomain coefficients and may even accelerate convergence. Sun et al [28] proposed a DCT-domain convolutional neural network in JPEG to learn the association between the reconstructed image and the original image, which effectively compensates for high-frequency information, thereby protecting the edge of the image.…”