DeepQTMT: A Deep Learning Approach for Fast QTMT-based CU Partition of Intra-mode VVC

Abstract

The latest standard Versatile Video Coding (VVC) significantly improves the coding efficiency over its ancestor standard High Efficiency Video Coding (HEVC), but at the expense of sharply increased complexity. In VVC, the quadtree plus multi-type tree (QTMT) structure of coding unit (CU) partition accounts for most of encoding time, due to the bruteforce search for recursive rate-distortion (RD) optimization. Instead of the brute-force QTMT search, this paper proposes a deep learning approach to predict the QTMT-based CU partition, for drastically accelerating the encoding process of intra-mode VVC. First, we establish a large-scale database containing sufficient CU partition patterns with diverse video content, which can facilitate the data-driven VVC complexity reduction. Next, we propose a multi-stage exit CNN (MSE-CNN) model with an early exit mechanism to determine the CU partition, in accord with the flexible QTMT structure at multiple stages. Then, we design an adaptive loss function for training the MSE-CNN model, synthesizing both the uncertain number of split modes and the target on minimized RD cost. Finally, a multi-threshold decision scheme is developed, achieving desirable trade-off between complexity and RD performance. Experimental results demonstrate that our approach can reduce the encoding time of VVC by 44.65%∼66.88% with the negligible Bjøntegaard delta bit-rate (BD-BR) of 1.322%∼3.188%, which significantly outperforms other state-of-the-art approaches.

Enabling Deep Learning for All-in EDGE paradigm

“…Early Exit [12,118,126,161,194,244,245,249,271,282,313] Model Selection [159,191,271,314] Result Cache [13,39,53,92,93,96,108,112,114,123,209,268,293,319] 3.3.1 Model Compression: Model compression techniques facilitate the deployment of resource-hungry AI models into resourceconstrained EDGE servers by reducing the complexity of the DNN. Model compression exploits the sparse nature of gradients' and computation involved while training the DNN model.…”

“…With the multiple early exit points, it can also be considered as an enabler for localized inference using shallow DNN models [271]. In [126], the authors proposed DeepQTMT to lower the encoding time spent on video compression. In the DeepQTMT, the authors utilised a multi-stage early exit mechanism to accommodate the high encoding time.…”

Analysis of VVC Intra Prediction Block Partitioning Structure

“…This high encoding effort boosted the development of some works focusing on reducing the VVC encoding time. Several of these works [14]- [21] focused on reducing the computational cost of the QTMT partitioning structure, especially for intra-frame prediction. These solutions include fast CU decisions based on statistical analysis [14]- [17] and machine learning techniques [18]- [21]; all these solutions focus only on luminance blocks.…”

“…Several of these works [14]- [21] focused on reducing the computational cost of the QTMT partitioning structure, especially for intra-frame prediction. These solutions include fast CU decisions based on statistical analysis [14]- [17] and machine learning techniques [18]- [21]; all these solutions focus only on luminance blocks. Lei et al [14] developed a fast solution to avoid unnecessary block partition evaluations, where a subset of directional intra-frame prediction modes is evaluated to estimate the horizontal and vertical partitioning cost of the current block.…”

“…Fu et al [18] presented a fast CU partitioning algorithm using a classifier based on the Bayesian decision rule; the information derived from the current CU and horizontal binary splitting is used as a model input feature. The works of Tissier et al [19], Zhao et al [20], and Li et al [21] proposed the use of a Convolutional Neural Network (CNN) to find the best block partitioning and reduce the encoding time.…”