Deep Learning-Based Luma and Chroma Fractional Interpolation in Video Coding

Abstract

Motion compensated prediction is one of the essential methods to reduce temporal redundancy in inter coding. The target of motion compensated prediction is to predict the current frame from the list of reference frames. Recent video coding standards commonly use interpolation filters to obtain sub-pixel for the best matching block located in the fractional position of the reference frame. However, the fixed filters are not flexible to adapt to the variety of natural video contents. Inspired by the success of Convolutional Neural Network (CNN) in super-resolution, we propose CNN-based fractional interpolation for Luminance (Luma) and Chrominance (Chroma) components in motion compensated prediction to improve the coding efficiency. Moreover, two syntax elements indicate interpolation methods for the Luminance and Chrominance components, have been added to bin-string and encoded by CABAC using regular mode. As a result, our proposal gains 2.9%, 0.3%, 0.6% Y, U, V BD-rate reduction, respectively, under low delay P configuration.INDEX TERMS Convolution neural network (CNN), fractional interpolation, video coding, motion compensated prediction.

Improved CNN-Based Learning of Interpolation Filters for Low-Complexity Inter Prediction in Video Coding

“…Lastly, CompetitionCNN's coding complexity for CTC and non-CTC sequences is outlined in Table IV. While prior approaches that had been proposed for HEVC required GPU coding environments [16], [17], [20], the proposed approach can be applied in conventional CPU-based codec environments, with tolerable complexity increases in both encoding and decoding time. In [8], a full three-layer network was implemented within VVC and demonstrated increases in encoding complexity by up to 380 times.…”

“…Application of linear segments has already been presented in the context of intra-prediction for video coding [22]. Furthermore, in previous work [8], a simplified CNN-based filtering process was proposed in the context of inter-prediction in VVC, as opposed to the approach in [14], [16], [20] which used HEVC as its baseline encoder. Due to the fact VVC is considerably more efficient than HEVC, it is a much more challenging baseline from which to achieve compression efficiency gains.…”

“…In the context of directly deriving the sub-pixel samples, an approach based on super-resolution CNNs to interpolate half-pixel samples was introduced in [14], bringing 0.9% Bjøntegaard delta-rate (BD-rate) [15] bit-rate reductions under low-delay P (LDP) configuration when replacing HEVC luma filters. Furthermore, training separate networks for luma and chroma channels was presented in [16]. The resulting models were tested within the HEVC reference software as a switchable interpolation filter, achieving 2.9% BD-rate coding gains under the LDP configuration.…”

Interpreting CNN For Low Complexity Learned Sub-Pixel Motion Compensation In Video Coding

“…An approach to using super-resolution CNNs to generate half-pixel interpolated fractional samples was introduced in [14], reporting 0.9% Bjntegaard delta-rate (BD-rate) [15] reductions under lowdelay P (LDP) configuration when replacing HEVC luma filters. Training separate networks for luma and chroma channels was presented in [16]. The resulting models were integrated within the HEVC reference software as a switchable interpolation filter, achieving 2.9% BD-rate coding gains under the LDP configuration.…”