Natural landmark tracking using triangle-based optical flow

Abstract

Tracking of manually defined natural landmarks in image sequences is a crucial task. The chosen landmarks are usually not related to "good features" in the sense of tracking methods. Thus, established tracking methods do not achieve accurate trajectories. Hence, in this contribution, a triangle-based optical flow approach is presented to stabilize the tracking of landmarks, which avoids drifting of the tracked landmarks. Triangulation is performed based on a defined ROI resulting in a mesh of triangles with uniform size. For each image frame, the optical flow is computed within the ROI. Based on the mean optical flow vector under each triangle area, the three corner points are shifted and realigned by averaging connected corners. Resulting in a deformation of the triangle mesh, a translation vector for the manually defined landmarks can be determined by interpolating the deformation of the corresponding area point of the relevant triangle. The proposed approach has been evaluated on intra-operative video recordings of bypass interventions in cardiac surgery, visualizing pre-operative angiography images as deformed overlays. While with standard methods the feature points are often lost after some frames or tend to drift in another than the real direction, the triangle-based optical flow stabilizes the drift of the feature points and leads to more accurate trajectories over a longer amount of time.