Layered Conceptual Image Compression Via Deep Semantic Synthesis

Chang, Jichuan; Mao, Qi; Zhao, Zhimin; Wang, Shiqi; Wang, Shiqi; Zhu, Hong; Ma, Siwei

doi:10.1109/icip.2019.8803805

Cited by 22 publications

(10 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [37], a generative decoder is adopted to map key frames as well as soft edges of non-key frames into the whole reconstructed frames. In [11], the edges are extracted to form the feature stream, which facilitates the image reconstruction at the decoder side.…”

Section: Feature Assisted Codingmentioning

confidence: 99%

Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

Yang¹,

Huang²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

As an emerging research practice leveraging recent advanced AI techniques, e.g. deep models based prediction and generation, Video Coding for Machines (VCM) is committed to bridging to an extent separate research tracks of video/image compression and feature compression, and attempts to optimize compactness and efficiency jointly from a unified perspective of high accuracy machine vision and full fidelity human vision. With the rapid advances of deep feature representation and visual data compression in mind, in this paper, we summarize VCM methodology and philosophy based on existing academia and industrial efforts. The development of VCM follows a general rate-distortion optimization, and the categorization of key modules or techniques is established including features assisted coding, scalable coding, intermediate feature compression/optimization, and machine vision targeted codec, from broader perspectives of vision tasks, analytics resources, etc. From previous works, it is demonstrated that, although existing works attempt to reveal the nature of scalable representation in bits when dealing with machine and human vision tasks, there remains a rare study in the generality of low bit rate representation, and accordingly how to support a variety of visual analytic tasks. Therefore, we investigate a novel visual information compression for the analytics taxonomy problem to strengthen the capability of compact visual representations extracted from multiple tasks for visual analytics. A new perspective of task relationships versus compression is revisited. By keeping in mind the transferability among different machine vision tasks (e.g. high-level semantic and mid-level geometry-related), we aim to support multiple tasks jointly at low bit rates. In particular, to narrow the dimensionality gap between neural network generated features extracted from pixels and a variety of machine vision features/labels (e.g. scene class, segmentation labels), a codebook hyperprior is designed to compress the neural network-generated features. As demonstrated in our experiments, this new hyperprior model is expected to improve feature compression efficiency by estimating the signal entropy more accurately, which enables further investigation of the granularity of abstracting compact features among different tasks.

show abstract

Section: Feature Assisted Codingmentioning

confidence: 99%

Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

Yang¹,

Huang²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In [6], a generative compression framework is proposed to encode an image into low-bit-rate latent code and exploit recurrent generative networks for reconstruction. With compressive variational auto-encoders (VAE) [10], generative networks are also utilized in [5] to reconstruct images from edges and latent features produced by neural networks. Though these frameworks encode compact feature representations of images, they are not shown to both satisfy the need of human and machine vision.…”

Section: Related Workmentioning

confidence: 99%

“…Some works show potential ways to address this problem. In [5,6], generative models are used to reconstruct the images based on the encoded features with very few bits towards conceptual coding. In [7], the bitstreams generated by a Variational Auto-Encoder (VAE) is used for image understanding.…”

Section: Introductionmentioning

confidence: 99%

Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach

Yang

et al. 2020

Preprint

View full text Add to dashboard Cite

The past decades have witnessed the rapid development of image and video coding techniques in the era of big data. However, the signal fidelity-driven coding pipeline design limits the capability of the existing image/video coding frameworks to fulfill the needs of both machine and human vision. In this paper, we come up with a novel image coding framework by leveraging both the compressive and the generative models, to support machine vision and human perception tasks jointly. Given an input image, the feature analysis is first applied, and then the generative model is employed to perform image reconstruction with features and additional reference pixels, in which compact edge maps are extracted in this work to connect both kinds of vision in a scalable way. The compact edge map serves as the basic layer for machine vision tasks, and the reference pixels act as a sort of enhanced layer to guarantee signal fidelity for human vision. By introducing advanced generative models, we train a flexible network to reconstruct images from compact feature representations and the reference pixels. Experimental results demonstrate the superiority of our framework in both human visual quality and facial landmark detection, which provide useful evidence on the emerging standardization efforts on MPEG VCM (Video Coding for Machine) 1 . Our project website is available at https://williamyang1991.github. io/projects/VCM-Face/.

show abstract

“…With the progress in deep generative models, conceptual coding [1,2,3] has emerged as a new paradigm for image compression beyond traditional signal-based image codecs, such as JPEG [4], JPEG2000 [5], HEVC [6] and other learningbased codecs [7,8,9]. Aiming at extracting decomposed conceptual representation from input visual data, conceptual coding not only achieves significant bitrate reduction over traditional codecs at comparable reconstruction quality, but also supports more flexible vision tasks.…”

Section: Introductionmentioning

confidence: 99%

“…However, the entropy modeling techniques are still under-explored for conceptual coding. Specifically, existing methods [2,3] typically adopt a structure-texture dual-layered framework, yet the acquired conceptual codes are often compressed without effective rate optimization. Furthermore, a single latent vector is usually leveraged to model global texture distribution of multiple semantic regions, where the intra-region similarity and cross-region independencies of texture codes are not fully exploited for entropy-constrained training.…”

Section: Introductionmentioning

confidence: 99%

Thousand to One: Semantic Prior Modeling for Conceptual Coding

Chang

Zhao

Yang

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

Self Cite

View full text Add to dashboard Cite

Conceptual coding has been an emerging research topic recently, which encodes natural images into disentangled conceptual representations for compression. However, the compression performance of the existing methods is still suboptimal due to the lack of comprehensive consideration of rate constraint and reconstruction quality. To this end, we propose a novel end-to-end semantic prior modeling based conceptual coding scheme towards extremely low bitrate image compression, which leverages semantic-wise deep representations as a unified prior for entropy estimation and texture synthesis. Specifically, we employ semantic segmentation maps as structural guidance for extracting deep semantic prior, which provides fine-grained texture distribution modeling for better detail construction and higher flexibility in subsequent highlevel vision tasks. Moreover, a cross-channel entropy model is proposed to further exploit the inter-channel correlation of the spatially independent semantic prior, leading to more accurate entropy estimation for rate-constrained training. The proposed scheme 1 achieves an ultra-high 1000× compression ratio, while still enjoying high visual reconstruction quality and versatility towards visual processing and analysis tasks.

show abstract

Layered Conceptual Image Compression Via Deep Semantic Synthesis

Cited by 22 publications

References 40 publications

Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach

Thousand to One: Semantic Prior Modeling for Conceptual Coding

Contact Info

Product

Resources

About