Keypoint Encoding for Improved Feature Extraction From Compressed Video at Low Bitrates

Abstract

In many mobile visual analysis applications, compressed video is transmitted over a communication network and analyzed by a server. Typical processing steps performed at the server include keypoint detection, descriptor calculation, and feature matching. Video compression has been shown to have an adverse effect on feature-matching performance. The negative impact of compression can be reduced by using the keypoints extracted from the uncompressed video to calculate descriptors from the compressed video. Based on this observation, we propose to provide these keypoints to the server as side information and to extract only the descriptors from the compressed video. First, we introduce four different frame types for keypoint encoding to address different types of changes in video content. These frame types represent a new scene, the same scene, a slowly changing scene, or a rapidly moving scene and are determined by comparing features between successive video frames. Then, we propose Intra, Skip and Inter modes of encoding the keypoints for different frame types. For example, keypoints for new scenes are encoded using the Intra mode, and keypoints for unchanged scenes are skipped. As a result, the bitrate of the side information related to keypoint encoding is significantly reduced. Finally, we present pairwise matching and image retrieval experiments conducted to evaluate the performance of the proposed approach using the Stanford mobile augmented reality dataset and 720p format videos. The results show that the proposed approach offers significantly improved feature matching and image retrieval performance at a given bitrate.

Video analytical coding: When video coding meets video analysis

“…video surveillance) where it is necessary to visualize the video content. By contrast, Chao and Steinbach [22] proposed a novel framework, in which keypoints extracted from a video were encoded and transmitted along with the compressed video. However, this framework needs more bits to transmit the feature descriptors.…”

“…In order to explore the impact of τ on the proposed framework, we test a set of values where τ ranges from 0.1 to 0.9. Four QP values (22,27,32,37) from PETS2009 [31]. Clip 1 to Clip 4 are captured by stationary cameras without any zooming.…”

Efficient High Quality Video Assessment Using Salient Features

“…However, heterogeneous devices consists of unified optimization problem in video encoding and provides high computation complexity. In [17], Key point Encoding for Improved Feature Extraction from Compressed Video at Low Bitrates performed to detect key-points and descriptor calculation, which helps in detecting different variations, occurs in video encoding, to lower bitrates. In this paper, they have reduced bitrate to a sufficient amount for a single scene of a video.…”

Analytical distortion aware video coding for computer based video analysis

“…In their later work [7], the coding architecture was applied to code binary local features. In order to decrease the adverse effect of video compression on feature-matching performance, Chao and Steinbach [8] proposed a novel framework, in which keypoints extracted from a video were encoded and transmitted along with the compressed video to the server. However, this framework needs more bits since the keypoints are transmitted as side information.…”