Bidirectional Mesh-Based Frame Rate Up-Conversion

Abstract

In this paper, we propose a new frame rate upconversion (FRUC) method for temporal video quality enhancement. The proposed algorithm generates an interpolated frame in between two given frames based on a bi-directional mesh interpolation (BMI), in order to cope with not only translation, but also with scale and rotation changes. BMI performance is highly influenced by the accuracy of the correspondences between the control points in the two frames. To achieve an accurate dense motion vector map (MVM) through bi-directional and unidirectional motion estimation, an initial MVM is formed by the transmitted motion vectors from coded bitstream with low computational complexity. Then, the interpolated frame is generated by frame-based BMI with the dense MVM. In our experiments, we found out that the proposed algorithm is about 2dB better than several conventional FRUC methods. Furthermore, block artifacts and blur artifacts are significantly diminished by the proposed algorithm. Index Terms-Frame rate up-conversion, mesh interpolation, uni-directional motion estimation, bi-directional motion estimation I. INTRODUCTIONrame rate up-conversion (FRUC) refers to the postprocessing methods whose goal is to improve video quality by increasing the frame rate. A number of FRUC algorithms have been developed [1]- [15], and they can be classified into two main categories. One category contains methods that generate interpolated frames without explicitly considering motion. These algorithms analyze mathematical relations regarding pixel intensity changes among neighboring frames. The second category includes motion-based techniques. Most of these methods generate interpolated frames with the help of block-based motion estimation and compensation [2]- [15]. These approaches are able to improve video quality with natural continuity of motion. However, the block-based translational motion model can lead to blocking artifacts.Mesh-based techniques [17], [18] are another alternative in an attempt to represent more realistic motion. Like most other interpolation methods, the mesh-based interpolation is sensitive to occlusions. In order to deal with these occlusions, the proposed method interpolates the target frame through bi-

Video frame interpolation via spatial multi‐scale modelling

“…The main issue of these pixel-based ME is the high computation complexity, and it is not suitable for real-time applications. Except for block-based and pixel-based ME, some traditional VFI methods also employed region-based [25], feature-based [26] or mesh-based ME [27], and the main idea was to classify pixels by considering the content in the video scene.…”