Coding Tree Unit Partition Prediction Algorithm Utilizing Residual AttentionBased Long Short-Term Memory for Video Intra Coding

Abstract

High-efficiency video coding (HEVC) stands as one of the most extensively employed video coding standards, with the primary encoding challenge revolving around the searching process. To minimize coding complexity of HEVC block partition, a new algorithm is proposed for supporting the partition architecture of coding tree unit (CTU) in intra-coding. In this research, problem of partition architecture decisions of CTU is resolved through two stages. In first phase, the bagged tree method is evaluated for predicting CTU splitting, and in the second phase, the partition issue of 32 × 32 size CU is given as 17 outcome classification. To attain a huge accuracy of prediction, a residual attention-based long short-term memory (LSTM) with parametric rectified linear unit (AT-LSTM-PReLU) is proposed. The proposed method produces a structure of partition quad-tree of CTU that makes multiple decisions at variant depth levels. The performance of proposed method is estimated with a dataset gathered from the joint collaborative team and video coding (JCT-VC). The proposed approach achieved a remarkable bit rate improvement of 3.72%, a significant ∆T enhancement of 97.28%, and minimized video quality degradation by just -0.05dB. These results outperform other deep learning methods like convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM).