Progressive Spatial Recurrent Neural Network for Intra Prediction

Abstract

Intra prediction is an important component of modern video codecs, which is able to efficiently squeeze out the spatial redundancy in video frames. With preceding pixels as the context, traditional intra prediction schemes generate linear predictions based on several predefined directions (i.e. modes) for blocks to be encoded. However, these modes are relatively simple and their predictions may fail when facing blocks with complex textures, which leads to additional bits encoding the residue. In this paper, we design a Progressive Spatial Recurrent Neural Network (PS-RNN) that learns to conduct intra prediction. Specifically, our PS-RNN consists of three spatial recurrent units and progressively generates predictions by passing information along from preceding contents to blocks to be encoded. To make our network generate predictions considering both distortion and bit-rate, we propose to use Sum of Absolute Transformed Difference (SATD) as the loss function to train PS-RNN since SATD is able to measure rate-distortion cost of encoding a residue block. Moreover, our method supports variable-block-size for intra prediction, which is more practical in real coding conditions. The proposed intra prediction scheme achieves on average 2.5% bit-rate reduction on variable-block-size settings under the same reconstruction quality compared with HEVC.

Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies

“…One would intuitively expect that coding performance can be further improved if better predictions can be produced. Therefore, there have been a number of attempts to leverage the powerful capacity of stacked DNNs for better intrapredictor generation, including the CNN-based predictor refinement suggested in [113] to reduce prediction residual, additional learned mode trained using FCN models reported in [114] and [115], using RNNs in [116], using CNNs in [108], even using GANs in [117], and so on. These approaches have actively utilized the neighbor pixels or blocks and/or other context information (e.g., mode) if applicable, in order to accurately represent the local structures for better prediction.…”

Merging Multiple Template Matching Predictions in Intra Coding with Attentive Convolutional Neural Network

“…In [16,17], a fully connected neural networks was used to construct the relation between boundary reconstructed pixels and original pixels in current block, and about 2%-4% bit-rate saving could be achieved. The researchers in [18] applied the recurrent neural network to progressively generate the prediction signal of current block, and improved the coding efficiency obviously. At almost the same time, the attention mechanism have become one hotspot in the literature [19][20][21][22].…”

Fully Neural Network Mode Based Intra Prediction of Variable Block Size

“…Dumas et al [5] stated that using convolutional neural network (CNN) performs better than FC for blocks larger than 8×8. Hu et al [6] presented a new structure based on recurrent neural network (RNN). Sun et al [7] studied different combination schemes of traditional modes (TM) and neural network modes (NM) for the fixed block 8×8.…”

“…First is that TM still remains in the coding framework. In [4], [5] and [6], one or two NMs were provided. In [7], at most seven NMs were exploited.…”