Robust GPU-assisted camera tracking using free-form surface models

Abstract

We propose a marker-less model-based camera tracking approach, which makes use of GPU-assisted analysis-by-synthesis methods on a very wide field of view (e.g. fish-eye) camera. After an initial registration based on a learned database of robust features, the synthesis part of the tracking is performed on graphics hardware, which simulates internal and external parameters of the camera, this way minimizing lens and viewpoint differences between a model view and a real camera image. Based on an automatically reconstructed free-form surface model we analyze the sensitivity of the tracking to the model accuracy, in particular the case when we represent curved surfaces by planar patches. We also examine accuracy and show on synthetic and on real data that the system does not suffer from drift accumulation. The wide field of view of the camera and the subdivision of our reference model into many textured free-form surface patches make the system robust against illumination changes, moving persons and other occlusions within the environment and provide a camera pose estimate in a fixed and known coordinate system.

Extraction of 3D freeform surfaces as visual landmarks for real-time tracking

“…From the presented results it can be seen that, although gaps around occlusions and texture-less regions occur in the models, the system performs well on textured scene parts, which are the most interesting regions for visual tracking in [31], for which the system is targeted. Fig.…”

“…Therefore, in order to be useful in a real-time analysis-by-synthesis camera tracking framework, the depth map is converted into a textured triangle mesh. This representation is undemanding in means of rendering facilities and hence can be rendered in real-time using commodity 3D accelerator cards, even when simulating spherical cameras [31] and using less powerful wearable computers.…”

“…The observed interpolated image values are robustly fused in a similar way as the depth values are and such give a texel's color. Thereby image noise and color quantization is reduced and the model texture quality for tracking in [31] is improved.…”

“…On the other hand, the FreeD system described in [61] does not have these problems but needs an expensively calibrated marker setup, which makes its application outside of prepared studio environments difficult. In order to overcome these limitations [30,31] propose a tracking system which relies on a 3D reference model that was extracted from an image sequence itself. It is proposed to split tracking into two phases: an offline phase where the model is generated and an online phase where the camera is tracked in real-time.…”

ToF-sensors: New dimensions for realism and interactivity

“…Figure 2 shows two sample scenes acquired with this kind of approach. For high quality reconstruction, the low resolution and small field of view of a ToF-camera can be compensated for by combining it with high-resolution imagebased 3D scene reconstruction, for example by utilizing a structure-from-motion (SFM) approach [1,16]. The inherent problem of SFM, that no metric scale can be obtained, is solved by the metric properties of the ToF-measurements.…”