Interframe Coding of Feature Descriptors for Mobile Augmented Reality

Abstract

Streaming mobile augmented reality applications require both real-time recognition and tracking of objects of interest in a video sequence. Typically, local features are calculated from the gradients of a canonical patch around a keypoint in individual video frames. In this paper, we propose a temporally coherent keypoint detector and design efficient interframe predictive coding techniques for canonical patches, feature descriptors, and keypoint locations. In the proposed system, we strive to transmit each patch or its equivalent feature descriptor with as few bits as possible by modifying a previously transmitted patch or descriptor. Our solution enables server-based mobile augmented reality where a continuous stream of salient information, sufficient for image-based retrieval, and object localization, is sent at a bit-rate that is practical for today's wireless links and less than one-tenth of the bit-rate needed to stream the compressed video to the server.

A survey on compact features for visual content analysis

“…According to such architecture, temporally coherent keypoints are detected, and the patches surrounding such keypoints are encoded in a predictive fashion [9]. Alternatively, descriptors can be encoded in place of patches in a predictive fashion [76]. The temporal correlation between video frames can be exploited to efficiently encode keypoint locations.…”

“…The temporal correlation between video frames can be exploited to efficiently encode keypoint locations. In particular, methods presented in Section III-B can be adapted to the context of keypoints extracted from video content [8,76].…”

“…Global encoding Bag-of-Words (BoW) [5], Pyramid Kernel [60], Tree codebook [61], Kernel codebook (KC) [62], Sparse coding [63], Locality-constrained Linear Coding (LLC) [64], Hamming Embedding (HE) [65], VLAD [66], Fisher Kernel (FK) [67], Super Vector [68], Bag-of-Binary-Words [69], BVLAD [70] Other Location coding [71,72], SIFT-Preserving JPEG [73] and H.264/AVC [74], Chen and Moulin [75], Hybrid ATC (HATC) [7], Interframe patch [9] and descriptor [76] coding, VideoSIFT [10], VideoBRISK [77] Feature networking Section V -Y a n g et al [78,79], feature extraction offloading [80][81][82], lossy feature transmission [3], Mobile Visual Search [83] exploited to encode visual features, providing a significant coding gain with respect to the case of still images. Similar works in the previous literatures focus on either feature extraction [12,13] or encoding [14,15].…”

GPS-Based Augmented Reality System of Past Topography Using Antique Maps

“…Also, if it is necessary for an administrative reason, as for the ancient/old books, the maps of photoprint-versions or traditional cartography can also be accounted as the antique maps [6][7][8][9][10][11][12][13][14][15]. In this paper, we define Korean antique maps as the ones that had been printed or copied with the traditional Korean production technique prior to 1910, excluding modern surveying technology.…”

Compact Global Descriptors for Visual Search

“…An alternate approach to sending JPEG images or MPEG videos is to extract feature descriptors on the mobile device, compress the descriptors and transmit them over the network. Such an approach has been demonstrated to reduce the amount of transmission data by orders of magnitude for both visual search and augmented reality applications [1], [2]. To this end, MPEG is close to completion of a standard titled Compact Descriptors for Visual Search (CDVS) for descriptor extraction and compression, and is embarking on extending the CDVS standard to video sources, titled Compact Descriptors for Video Analysis (CDVA).…”