Deep Reference Generation With Multi-Domain Hierarchical Constraints for Inter Prediction

Liu, Jiaying; Xia, Sifeng; Yang, Wenhan

doi:10.1109/tmm.2019.2961504

Cited by 27 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhao et al 17 proposed a CNN-based enhancement model to replace the traditional frame rate up conversion (FRUC) algorithm in both frame and coding tree unit (CTU) level. Liu et al 18 proposed a multi-scale quality attentive factorized kernel convolutional neural network (MQ-FKCNN) to generate additional reference frames. Huo et al 19 have proposed a deep network-based frame extrapolation method using reference frame alignment for HEVC and VVC.…”

Section: Deep Learning-based Inter Prediction In Video Codingmentioning

confidence: 99%

Attention-based Bi-Prediction Network with Knowledge Distillation for Versatile Video Coding (VVC)

Choi

Lee

Kim

et al. 2022

Preprint

View full text Add to dashboard Cite

In video coding, inter bi-prediction contributes to improve the coding efficiency significantly by producing precise fused prediction block. Although block-wise bi-prediction methods, such as bi-prediction with CU-level weight (BCW), are applied in Versatile Video Coding (VVC), linear fusion-based strategy is still difficult to represent diverse pixel variations inside a block. In addition, a pixel-wise bi-prediction method called bi-directional optical flow (BDOF) has been proposed to refine bi-prediction block. However, non-linear optical flow equation in BDOF mode is applied under assumptions so this method is yet unable to accurately compensate various kind of bi-prediction block. In this paper, we propose an attention-based bi-prediction network (ABPN) to substitute the whole existing bi-prediction methods. The proposed ABPN is designed to learn efficient representations of the fused features by utilizing attention mechanism. Furthermore, the knowledge distillation (KD)-based approach is employed to compress the size of the proposed network while keeping comparable output as the large model. The proposed ABPN is integrated into the VTM-11.0 NNVC-1.0 standard reference software. When compared with VTM anchor, it is verified that the BD-rate reduction of the lightweighted ABPN can be up to 5.89\% and 4.91\% on Y component under random access (RA) and low delay B (LDB), respectively.

show abstract

Section: Deep Learning-based Inter Prediction In Video Codingmentioning

confidence: 99%

Attention-based Bi-Prediction Network with Knowledge Distillation for Versatile Video Coding (VVC)

Choi

Lee

Kim

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Many previous works have showed the possibility of rate-distortion (RD) performance improvement by using deep learning in the subprocess during coding, e.g., block up-sampling [10], in-loop filtering [11], fractional interpolation [12], deep reference interpolation of B frame [13], etc. Liu et al proposed a deep frame interpolation network [14] to generate additional deep reference pictures for B frames.…”

Section: Introductionmentioning

confidence: 99%

Dilated convolutional neural network-based deep reference picture generation for video compression

Tian¹,

Gao²,

Wei³

et al. 2022

Preprint

View full text Add to dashboard Cite

Motion estimation and motion compensation are indispensable parts of inter prediction in video coding. Since the motion vector of objects is mostly in fractional pixel units, original reference pictures may not accurately provide a suitable reference for motion compensation. In this paper, we propose a deep reference picture generator which can create a picture that is more relevant to the current encoding frame, thereby further reducing temporal redundancy and improving video compression efficiency. Inspired by the recent progress of Convolutional Neural Network(CNN), this paper proposes to use a dilated CNN to build the generator. Moreover, we insert the generated deep picture into Versatile Video Coding(VVC) as a reference picture and perform a comprehensive set of experiments to evaluate the effectiveness of our network on the latest VVC Test Model-VTM. The experimental results demonstrate that our proposed method achieves on average 9.7% bit saving compared with VVC under low-delay P configuration.

show abstract

“…In early researches [3][4][5], their models are not capable enough to consider beyond their own purposes (only for one of human vision or machine vision). Even in the recent decade, most of the enhancement methods, e.g., dehazing [6][7][8][9], deraining [10][11][12][13][14][15][16][17][18][19], illumination enhancement [20][21][22][23][24][25], image/video compression [26][27][28], compression artifact removal [29,30], and copymove forgery detection [31], only target human vision and most of image/video understanding and analytics methods, e.g. classification [32], segmentation [33], action recognition [34], and human pose estimation [35], are considered to take the clean and high-quality images as the input of a system.…”

Section: Introductionmentioning

confidence: 99%

Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study

Yang

Yuan

Ren

et al. 2020

IEEE Trans. on Image Process.

Self Cite

189

View full text Add to dashboard Cite

Existing enhancement methods are empirically expected to help the high-level end computer vision task: however, that is observed to not always be the case in practice. We focus on object or face detection in poor visibility enhancements caused by bad weathers (haze, rain) and low light conditions. To provide a more thorough examination and fair comparison, we introduce three benchmark sets collected in real-world hazy, rainy, and lowlight conditions, respectively, with annotated objects/faces. We launched the UG 2+ challenge Track 2 competition in IEEE CVPR 2019, aiming to evoke a comprehensive discussion and exploration about whether and how low-level vision techniques can benefit the high-level automatic visual recognition in various scenarios. To our best knowledge, this is the first and currently largest effort of its kind. Baseline results by cascading existing enhancement and detection models are reported, indicating the highly challenging nature of our new data as well as the large room for further technical innovations. Thanks to a large participation from the research community, we are able to analyze representative team solutions, striving to better identify the strengths and limitations of existing mindsets as well as the future directions.Index Terms-Poor visibility environment, object detection, face detection, haze, rain, low-light conditions *The first two authors Wenhan Yang and Ye Yuan contributed equally. Ye Yuan and Wenhan Yang helped prepare the dataset proposed for the UG2+ Challenges, and were the main responsible members for UG2+ Challenge 2019 (Track 2) platform setup and technical support. Wenqi Ren, Jiaying Liu, Walter J. Scheirer, and Zhangyang Wang were the main organizers of the challenge and helped prepare the dataset, raise sponsors, set up evaluation environment, and improve the technical submission. Other authors are the group members of winner teams in UG2+ challenge Track 2 contributing to the winning methods.

show abstract

Deep Reference Generation With Multi-Domain Hierarchical Constraints for Inter Prediction

Cited by 27 publications

References 46 publications

Attention-based Bi-Prediction Network with Knowledge Distillation for Versatile Video Coding (VVC)

Attention-based Bi-Prediction Network with Knowledge Distillation for Versatile Video Coding (VVC)

Dilated convolutional neural network-based deep reference picture generation for video compression

Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study

Contact Info

Product

Resources

About