A novel augmented reality framework based on monocular semi‐dense simultaneous localization and mapping

Abstract

Markerless tracking has been a trend in augmented reality (AR) applications nowadays, but it no longer satisfies users who want virtual characters to interact with the real world such as collision. Some sparse or dense simultaneous localization and mapping (SLAM) methods are proposed aiming to solve this problem. However, sparse methods only extract a plane from the sparse map, which cannot allow virtual characters to move realistically. Meanwhile, dense methods usually require powerful graphics processing unit (GPU) for dense mapping. In this paper, we present a real-time AR framework based on a semi-dense method with central processing unit (CPU). Specifically, the semi-dense method searches pixels with high gradients in each keyframe and estimates accurate depths by fusing matching pixels in other keyframes. We propose an outlier removal method that excludes three-dimensional points outside the camera trajectory. By integrating this method, our framework preserves clean edges of the real environment. The experimental results on the dataset show that our proposed framework has better surface reconstruction accuracy than other methods and our tracking thread runs in an acceptable speed when the semi-dense mapping thread runs backend. With the benefit of the robust camera tracking and the aligned surface, virtual characters of our AR application enable realistic movement and collision. K E Y W O R D SAR, monocular SLAM, semi-dense mapping, surface reconstruction