Augmented Reality Marker Hiding with Texture Deformation

Abstract

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

“…Most of existing work assumes that the background is planar [9,10,21] because image inpainting is potentially a screen space image processing. The recent work can handle curved [22] or multiple surfaces [23].…”

“…Almost all of the existing DR methods allow six degrees of freedom (6DoF) motion for the main camera using positioning sensors [33,34], fiducial markers [22,25,41,49], model-based tracking [13,26], vSLAM [14,23,27,35,44,53], etc. Cosco et al used ARToolKit and ARToolKitPlus markers to track the main camera at pre-observation and run-time for indoor desktop a DR scenario [25,49].…”

“…Herling and Broll used an object tracker that continuously detects and tracks the ROI to fill in the ROI with pixels using inpaint [9,10]. If the target object to be diminished is a marker [21,22,38] or a marker-attached object [31], then the fiducial markers are used to estimate the camera pose.…”

“…When a target object is a plane marker, the ROI can be automatically determined from the marker information [21,22,38]. Hasegawa and Saito used HOG-SVM [61] and Kalman filter [62] to automatically determine an area of a moving person to be eliminated [16,17].…”

“…Kawai et al proposed a method for segmenting scenes into multiple planes using a point cloud reconstructed with a vSLAM and executed image inpainting on each plane [23]. They also extended an imageinpainting algorithm to inpaint a marker on a deformed surface [22].…”