Interpreting CNN For Low Complexity Learned Sub-Pixel Motion Compensation In Video Coding

Murn, Luka; Blasi, Saverio; Smeaton, Alan F.; O’Connor, Noel E.; Mrak, Marta

doi:10.1109/icip40778.2020.9191193

Cited by 13 publications

(14 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methods based on Convolutional Neural Networks (CNNs) [2], [4] provided significant improvements at the cost of two main drawbacks: the associated increase in system complexity and the tendency to disregard the location of individual reference samples. Related works deployed complex neural networks (NNs) by means of model-based interpretability [5]. For instance, VVC recently adopted simplified NN-based methods such as Matrix Intra Prediction (MIP) modes [6] and Low-Frequency Non Separable Transform (LFNST) [7].…”

Section: Introductionmentioning

confidence: 99%

Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding

Blanch¹,

Blasi²,

Smeaton

et al. 2021

IEEE J. Sel. Top. Signal Process.

Self Cite

View full text Add to dashboard Cite

Neural networks can be successfully used to improve several modules of advanced video coding schemes. In particular, compression of colour components was shown to greatly benefit from usage of machine learning models, thanks to the design of appropriate attention-based architectures that allow the prediction to exploit specific samples in the reference region. However, such architectures tend to be complex and computationally intense, and may be difficult to deploy in a practical video coding pipeline. This work focuses on reducing the complexity of such methodologies, to design a set of simplified and cost-effective attention-based architectures for chroma intra-prediction. A novel size-agnostic multi-model approach is proposed to reduce the complexity of the inference process. The resulting simplified architecture is still capable of outperforming state-of-the-art methods. Moreover, a collection of simplifications is presented in this paper, to further reduce the complexity overhead of the proposed prediction architecture. Thanks to these simplifications, a reduction in the number of parameters of around 90% is achieved with respect to the original attentionbased methodologies. Simplifications include a framework for reducing the overhead of the convolutional operations, a simplified cross-component processing model integrated into the original architecture, and a methodology to perform integer-precision approximations with the aim to obtain fast and hardware-aware implementations. The proposed schemes are integrated into the Versatile Video Coding (VVC) prediction pipeline, retaining compression efficiency of state-of-the-art chroma intra-prediction methods based on neural networks, while offering different directions for significantly reducing coding complexity.

show abstract

Section: Introductionmentioning

confidence: 99%

Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding

Blanch¹,

Blasi²,

Smeaton

et al. 2021

IEEE J. Sel. Top. Signal Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Given the successful application of ML to super-resolution applications, neural networks may be employed within a video coding scheme to produce new, more efficient interpolation filters. In a previous work [8], an approach for CNN-based sub-pixel interpolation filters for fractional motion estimation was presented in the context of the VVC standard. This is the first approach that shows gains in the challenging VVC framework, while also reducing the computational requirements to an acceptable level.…”

Section: Introductionmentioning

confidence: 99%

“…This is the first approach that shows gains in the challenging VVC framework, while also reducing the computational requirements to an acceptable level. This paper expands on such work [8], uncovering new ways to boost the prediction performance of the learned interpolation, without affecting the computational requirements of the prediction process. In particular this paper introduces:…”

Section: Introductionmentioning

confidence: 99%

“…While these approaches provide powerful prediction performance thanks to NN-generated sub-pixel samples, they have very high computational requirements, increasing the encoding complexity by up to 380 times [8], which makes them inapplicable in real world scenarios. Furthermore, these NN-based interpolation filters are still used as "black-boxes", i.e.…”

Section: Introductionmentioning

confidence: 99%

“…Application of linear segments has already been presented in the context of intra-prediction for video coding [22]. Furthermore, in previous work [8], a simplified CNN-based filtering process was proposed in the context of inter-prediction in VVC, as opposed to the approach in [14], [16], [20] which used HEVC as its baseline encoder. Due to the fact VVC is considerably more efficient than HEVC, it is a much more challenging baseline from which to achieve compression efficiency gains.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improved CNN-Based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding

Murn¹,

Blasi²,

Smeaton

et al. 2021

IEEE Open J. Signal Process.

Self Cite

View full text Add to dashboard Cite

The versatility of recent machine learning approaches makes them ideal for improvement of next generation video compression solutions. Unfortunately, these approaches typically bring significant increases in computational complexity and are difficult to interpret into explainable models, affecting their potential for implementation within practical video coding applications. This paper introduces a novel explainable neural network-based inter-prediction scheme, to improve the interpolation of reference samples needed for fractional precision motion compensation. The approach requires a single neural network to be trained from which a full quarter-pixel interpolation filter set is derived, as the network is easily interpretable due to its linear structure. A novel training framework enables each network branch to resemble a specific fractional shift. This practical solution makes it very efficient to use alongside conventional video coding schemes. When implemented in the context of the state-of-the-art Versatile Video Coding (VVC) test model, 0.77%, 1.27% and 2.25% BD-rate savings can be achieved on average for lower resolution sequences under the random access, low-delay B and low-delay P configurations, respectively, while the complexity of the learned interpolation schemes is significantly reduced compared to the interpolation with full CNNs.

show abstract