Real-Time Object Tracking for Augmented Reality Combining Graph Cuts and Optical Flow

Abstract

Figure 1: Tracking and annotating an object using graph cut segmentation. A building sign on the USC campus is first detected using simple recognition, after which no additional information is needed. As the camera moves, we segment and track the sign through significant scale and orientation changes, rendering an annotation above it. This example illustrates a possible navigation tool for exploring buildings on campus. In this case, the annotation tells the user that the building is Powell Hall, and that the offices inside include the CGIT Lab, the IMSC offices, and the EE department. ABSTRACTWe present an efficient and accurate object tracking algorithm based on the concept of graph cut segmentation. The ability to track visible objects in real-time provides an invaluable tool for the implementation of markerless Augmented Reality. Once an object has been detected, it's location in future frames can be used to position virtual content, and thus annotate the environment.Unlike many object tracking algorithms, our approach does not rely on a preexisting 3D model or any other information about the object or its environment. It takes, as input, a set of pixels representing an object in an initial frame and uses a combination of optical flow and graph cut segmentation to determine the corresponding pixels in each future frame. Experiments show that our algorithm robustly tracks objects of disparate shapes and sizes over hundreds of frames, and can even handle difficult cases where an object contains many of the same colors as its background. We further show how this technology can be applied to practical AR applications.

Forked! A demonstration of physics realism in augmented reality

“…Planned directions for future work include: the use of nonfiducial based registration methods (e.g. [4]; the use of more sophisticated rendering techniques and possibly 3D viewing; making use of advanced features of the physics engine; and allowing virtual objects to physically impact real ones (similarly to [1]).…”

Accelerating Live Graph-Cut-Based Object Tracking Using CUDA

“…By utilizing a dilation of the previous graph cut's resultant binary image, the operations of function graph cuts ca be masked, resulting in a smaller graph on which to operate. Similar methods which use banded graph cuts, cutting only around the boundary of the object, have been proposed Lambaert et al (2005); Mooser et al (2007), however these methods have been shown to lose information and misclassify pixels in certain topologies Garrett & Saito (2008); Sinop & Grady (2006). Initial timing analysis of graph cuts shown in Fig.…”

Interactive modelling for AR applications

“…This method produces good results and does not require a pre-existing model, but it does not run in real-time. Mooser et al [11] have built upon Lepetit and Berger's approach but use optical flow to transfer an object boundary into a new image.…”