Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding

Abstract

Neural networks can be successfully used to improve several modules of advanced video coding schemes. In particular, compression of colour components was shown to greatly benefit from usage of machine learning models, thanks to the design of appropriate attention-based architectures that allow the prediction to exploit specific samples in the reference region. However, such architectures tend to be complex and computationally intense, and may be difficult to deploy in a practical video coding pipeline. This work focuses on reducing the complexity of such methodologies, to design a set of simplified and cost-effective attention-based architectures for chroma intra-prediction. A novel size-agnostic multi-model approach is proposed to reduce the complexity of the inference process. The resulting simplified architecture is still capable of outperforming state-of-the-art methods. Moreover, a collection of simplifications is presented in this paper, to further reduce the complexity overhead of the proposed prediction architecture. Thanks to these simplifications, a reduction in the number of parameters of around 90% is achieved with respect to the original attentionbased methodologies. Simplifications include a framework for reducing the overhead of the convolutional operations, a simplified cross-component processing model integrated into the original architecture, and a methodology to perform integer-precision approximations with the aim to obtain fast and hardware-aware implementations. The proposed schemes are integrated into the Versatile Video Coding (VVC) prediction pipeline, retaining compression efficiency of state-of-the-art chroma intra-prediction methods based on neural networks, while offering different directions for significantly reducing coding complexity.

Open-Source: Attention-Based Neural Networks For Chroma Intra Prediction In Video Coding

“…The attention-based model proposed in [2], adopts a scheme based on three network branches that are combined to produce prediction samples. The first branch (cross-component boundary branch) extracts cross component features and encodes the colours on the boundary.…”

“…Finally, the combined features are transformed to actual colours using a third convolutional branch (the prediction head). The multi-model architecture proposed in [2], improves the baseline attention-based approach. Main improvements are associated to its block-size agnostic property as the proposed approach only requires one model for all block sizes.…”

“…This software 1 focuses on complexity reduction, implementing the collection of simplified attention-based architectures proposed in [2]. A novel training methodology is proposed in order to design a block-independent multi-model which outperforms the state-of-the-art attention-based architectures and reduces inference complexity.…”

New VVC Chroma Prediction Modes Based on Coloring with Inter-Channel Correlation

“…Blanch et al [9] presented a neural network architecture for cross component intra prediction, in which an attention module was employed for learning spatial relations. In [10], the authors further improved the model and reduce the complexity. Zhu et al [11] presented a new neural network architecture for chroma prediction.…”

Improved CNN-Based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding

“…Block diagram of a modern video encoder (e.g. VVC [7]), with highlighted inter-prediction modules compression tasks such as chroma intra-prediction [6], with very promising results.…”