Learning based Facial Image Compression with semantic fidelity metric

Abstract

Surveillance and security scenarios usually require high efficient facial image compression scheme for face recognition and identification. While either traditional general image codecs or special facial image compression schemes only heuristically refine codec separately according to face verification accuracy metric. We propose a Learning based Facial Image Compression (LFIC) framework with a novel Regionally Adaptive Pooling (RAP) module whose parameters can be automatically optimized according to gradient feedback from an integrated hybrid semantic fidelity metric, including a successfully exploration to apply Generative Adversarial Network (GAN) as metric directly in image compression scheme. The experimental results verify the framework's efficiency by demonstrating performance improvement of 71.41%, 48.28% and 52.67% bitrate saving separately over JPEG2000, WebP and neural network-based codecs under the same face verification accuracy distortion metric. We also evaluate LFIC's superior performance gain compared with latest specific facial image codecs. Visual experiments also show some interesting insight on how LFIC can automatically capture the information in critical areas based on semantic distortion metrics for optimized compression, which is quite different from the heuristic way of optimization in traditional image compression algorithms.

Visual Analysis Motivated Rate-Distortion Model for Image Coding

“…Rate-distortion optimization (RDO) plays an important role in improving performance in image coding. According to [3], types of distortions are classified into three categories: pixel level distortion, perceptual distortion for human vision, and semantic distortion for machine vision. The most commonly used pixel fidelity metric is mean square error (MSE).…”

“…characteristics of the machine vision system (MVS) to encode visual analysis-friendly image/video becomes an essential challenge. Chen et al [3] propose a Regionally Adaptive Pooling (RAP) module which could apply Generative Adversarial Network (GAN) as metric directly in image compression scheme to improve performance on facial analysis tasks. Shi et al [16] combine reinforcement learning with High Efficiency Video Coding (HEVC) to determine QP based on prior knowledge of the target visual task.…”

Multi-scale Grouped Dense Network for VVC Intra Coding

“…With the development of deep neural network, learning based image compression algorithms [1,7,8,2,16,3,4] have drawn huge attention. Belie et al [1] implemented the end-to-end optimized image compression by utilizing the general divisive normalization and soft quantization with additive uniform noise.…”

Reinforced Bit Allocation under Task-Driven Semantic Distortion Metrics

“…However, the problem is how to define the distortion. Generally, it can be classified into three levels of distortion metrics: Pixel Fidelity, Perceptual Fidelity, and Semantic Fidelity, according to different levels of human cognition on image/video signal [1].…”

“…Some approaches [16], [17] tried to do this by adjusting image compression parameters (e.g., quantization parameters (QPs)) heuristically according to the embedded quality metrics, but they are still heuristic solutions without ability to automatically and adaptively optimize encoding configurations according to different complicated distortion metrics. [1] is the first scheme trying to solve this challenge by designing an end-to-end image coding framework which inherently provides feasibility on integrating complicated metrics into coding loop. But its still a huge challenge for traditional hybrid coding frameworks which are not derivable like end-to-end image coding framework.…”