Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Abstract

In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression and headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation and head translation), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the stateof-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse. The project page can be found at https://github.com/Berlin0610/Interactive Face Video Coding.

UGC-VIDEO: Perceptual Quality Assessment of User-Generated Videos

“…• CFTE [16]: GAN-based model with deep learning based feature representation. Each reference video is compressed by these codecs to three to five compression levels.…”

“…In [47], it shows that DISTS and LPIPS deliver average-level yet the best performance on generative image compression among all IQA methods. Though the prediction accuracy is unsatisfactory, the two measures are still used to evaluate the performance of generative face video compression [16] because of the computational efficiency. Video quality assessment involves the temporal domain which significantly influences perceptual quality.…”

“…The First Order Motion Model (FOMM) [14] was proposed for face image animation, based on which Oquab et al proposed [15] for mobile video chat scenarios. Moreover, Chen et al [16] proposed an E2E talking-head video compression framework (CFTE) via compact feature learning, achieving high compression efficiency under ultra-low bit rate scenarios. These face-specific video coding methods targeting low bit-rate have been claimed to be efficient in providing much better visual quality at the same level of coding bits.…”

“…2) Objective Video Quality Assessment: The FR-VQA/IQA algorithms which quantify image quality by measuring the similarity between the reference and distorted videos, have been widely applied in video compression performance evaluation and optimization [5], [6], [16], [35]. The most popular method peak signal-to-noise ratio (PSNR) has been widely criticized due to the low correlation with the HVS [36].…”

Dynamic Multi-Reference Generative Prediction for Face Video Compression

“…The First Motion Order Model (FOMM) [7] proposed to describe the face based on keypoints and this framework was further investigated in [8][9]. In [10], the face evolution was extracted through the compact features. In Face-vid2vid [11], the head motion is encoded based on a novel keypoint representation where the identity-specific and motion-related information is decomposed using unsupervised method.…”