The hierarchical quadtree partitioning of Coding Tree Units (CTU) is one of the striking features in HEVC that contributes towards its superior coding performance over its predecessors. However, the brute force evaluation of the quadtree hierarchy using the Rate-Distortion (RD) optimisation, to determine the best partitioning structure for a given content, makes it one of the most time-consuming operations in HEVC encoding. In this context, this paper proposes an intelligent fast Coding Unit (CU) size selection algorithm to expedite the encoding process of HEVC inter-prediction. The proposed algorithm introduces (i) two CU split likelihood modelling and classification approaches using Support Vector Machines (SVM) and Bayesian probabilistic models, and (ii) a fast CU selection algorithm that makes use of both offline trained SVMs and online trained Bayesian probabilistic models. Finally, (iii) a computational complexity to coding efficiency trade-off mechanism is introduced to flexibly control the algorithm to suit different encoding requirements. The experimental results of the proposed algorithm demonstrate an average encoding time reduction performance of 53.46%, 61.15%, and 58.15% for Low Delay B, Random Access, and Low Delay P configurations, respectively, with Bjøntegaard Delta-Bit Rate (BD-BR) losses of 2.35%, 2.9%, and 2.35%, respectively, when evaluated across a wide range of content types and quality levels. INDEX TERMS Coding Unit (CU), encoder complexity reduction, High Efficiency Video Coding (HEVC), inter-prediction, Support Vector Machine (SVM) R ECENT advancements in multimedia technologies that span across Consumer Electronics (CE) in video content capturing, transmission and display have made video data the most frequently exchanged type of content over the modern communication networks. The increasing mobile consumption of High Definition (HD) and Ultra High Definition (UHD) video contents has contributed immensely towards the ever-growing IP video traffic and it is expected to reach over 82% of the overall Internet traffic in 2021 [1]. However, the estimated growth in network bandwidth (1.9 fold from 2017-2022, which is 39.0 Mbps to 75.4 Mbps for fixed broadband [1]) with time is still not sufficient to cater for the ever-growing user demands. Furthermore, the video requirements for emerging applications such as Augmented Reality (AR)/Virtual Reality (VR), interactive television, multi-party video conferences and over-the-top (OTT) multimedia consumption demand continuous improvements in the compression efficiency [2]. In this regard, High Efficiency Video Coding (HEVC) which was introduced in 2013 is the most recent stable video coding standard. It provides greater compression efficiency through an assortment of new features and coding tools over its predecessor H.264/AVC [3]. Out of these, the hierarchical quadtree partitioning structure introduced in HEVC that entails a wide range of Coding Unit (CU) sizes (i.e., 8 × 8 to 64 × 64) and their combinations, is one of the important contributors...