Reducing Complexity of HEVC: A Deep Learning Approach

Xu, Mai; Li, Tianyi; Wang, Zulin; Deng, Xin; Yang, Ren; Guan, Zhenyu

doi:10.1109/tip.2018.2847035

Cited by 291 publications

(179 citation statements)

References 34 publications

Supporting

Mentioning

143

Contrasting

Order By: Relevance

“…Recently, DNN has shown splendid achievement in many research areas, e.g., video coding [37] and computer vision [38,39]. Also, several DNN approaches were proposed to estimate image memorability, which significantly improve the prediction accuracy.…”

Section: Related Workmentioning

confidence: 99%

What Makes Natural Scene Memorable?

Yang

et al. 2018

Proceedings of the 2018 Workshop on Understanding Subjective Attributes of Data, With the Focus on Evoked Emotions

Self Cite

View full text Add to dashboard Cite

Recent studies on image memorability have shed light on the visual features that make generic images, object images or face photographs memorable. However, a clear understanding and reliable estimation of natural scene memorability remain elusive. In this paper, we provide an attempt to answer: "what exactly makes natural scene memorable". Specifically, we first build LNSIM, a large-scale natural scene image memorability database (containing 2,632 images and memorability annotations). Then, we mine our database to investigate how low-, middle-and high-level handcrafted features affect the memorability of natural scene. In particular, we find that high-level feature of scene category is rather correlated with natural scene memorability. Thus, we propose a deep neural network based natural scene memorability (DeepNSM) predictor, which takes advantage of scene category. Finally, the experimental results validate the effectiveness of DeepNSM.

show abstract

Section: Related Workmentioning

confidence: 99%

What Makes Natural Scene Memorable?

Yang

et al. 2018

Proceedings of the 2018 Workshop on Understanding Subjective Attributes of Data, With the Focus on Evoked Emotions

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, only a few work has been done on inter coding. In [183], Convolutional neural network (CNN) and long-and shortterm memory (LSTM) network are utilized to predict the CU partition for both intra-/inter-modes to reduce HEVC complexity.…”

Section: Finer Precision Motion Estimation and Compensationmentioning

confidence: 99%

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

Zhang

Fan

et al. 2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

High Efficiency Video Coding (HEVC) has doubled the video compression ratio with equivalent subjective quality as compared to its predecessor H.264/AVC. The significant coding efficiency improvement is attributed to many new techniques. Interframe coding is one of the most powerful yet complicated techniques therein and has posed high computational burden thus main obstacle in HEVC-based real-time applications. Recently, plenty of research has been done to optimize the inter-frame coding, either to reduce the complexity for real-time applications, or to further enhance the encoding efficiency. In this paper, we provide a comprehensive review of the state-of-the-art techniques for HEVC inter-frame coding from three aspects, namely fast inter coding solutions, implementation on different hardware platforms as well as advanced inter coding techniques. More specifically, different algorithms in each aspect are further subdivided into sub-categories and compared in terms of pros, cons, coding efficiency and coding complexity. To the best of our knowledge, this is the first such comprehensive review of the recent advances of the inter-frame coding for HEVC and hopefully it would help the improvement, implementation and applications of HEVC as well as the ongoing development of the next generation video coding standard.

show abstract

“…In addition to improving loop filtering performance, Laude and Ostermann [30] proposed to replace the conventional Rate Distortion Optimization (RDO) with CNN for the intra prediction mode decision. Furthermore, coding unit (CU) partition mode decision can also be predicted by CNN [31] and Long and Short-Term Memory (LSTM) network [32].…”

Section: B Deep Learning-based Video Codingmentioning

confidence: 99%

Deep Reference Generation With Multi-Domain Hierarchical Constraints for Inter Prediction

Liu

Xia

Yang

2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Inter prediction is an important module in video coding for temporal redundancy removal, where similar reference blocks are searched from previously coded frames and employed to predict the block to be coded. Although traditional video codecs can estimate and compensate for block-level motions, their inter prediction performance is still heavily affected by the remaining inconsistent pixel-wise displacement caused by irregular rotation and deformation. In this paper, we address the problem by proposing a deep frame interpolation network to generate additional reference frames in coding scenarios. First, we summarize the previous adaptive convolutions used for frame interpolation and propose a factorized kernel convolutional network to improve the modeling capacity and simultaneously keep its compact form. Second, to better train this network, multi-domain hierarchical constraints are introduced to regularize the training of our factorized kernel convolutional network. For spatial domain, we use a gradually down-sampled and upsampled auto-encoder to generate the factorized kernels for frame interpolation at different scales. For quality domain, considering the inconsistent quality of the input frames, the factorized kernel convolution is modulated with quality-related features to learn to exploit more information from high quality frames. For frequency domain, a sum of absolute transformed difference loss that performs frequency transformation is utilized to facilitate network optimization from the view of coding performance. With the well-designed frame interpolation network regularized by multi-domain hierarchical constraints, our method surpasses HEVC on average 6.1% BD-rate saving and up to 11.0% BD-rate saving for the luma component under the random access configuration.

show abstract

Reducing Complexity of HEVC: A Deep Learning Approach

Cited by 291 publications

References 34 publications

What Makes Natural Scene Memorable?

What Makes Natural Scene Memorable?

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

Deep Reference Generation With Multi-Domain Hierarchical Constraints for Inter Prediction

Contact Info

Product

Resources

About