Adaptive user perspective rendering for Handheld Augmented Reality

Abstract

Handheld Augmented Reality commonly implements some variant of magic lens rendering, which turns only a fraction of the users real environment into AR while the rest of the environment remains unaffected. Since handheld AR devices are commonly equipped with video see-through capabilities, AR magic lens applications often suffer from spatial distortions, because the AR environment is presented from the perspective of the camera of the mobile device. Recent approaches counteract this distortion based on estimations of the users head position, rendering the scene from the user's perspective. To this end, approaches usually apply face-tracking algorithms on the front camera of the mobile device. However, this demands high computational resources and therefore commonly affects the performance of the application beyond the already high computational load of AR applications. In this paper, we present a method to reduce the computational demands for user perspective rendering by applying lightweight optical flow tracking and an estimation of the users motion before head tracking is started. We demonstrate the suitability of our approach for computationally limited mobile devices and we compare it to device perspective rendering, to head tracked user perspective rendering, as well as to fixed point of view user perspective rendering.

Don’t Bother Me: How to Handle Content-Irrelevant Objects in Handheld Augmented Reality

“…Another perceptual issue common in handheld AR is dual-view problem [22]. Dual-view problem relates to the difference in viewpoints between a user's eye and a device's camera.…”

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

“…Another problem, inherent to the relatively small handheld displays, is the dual views problem [15]-due to the limited size of the display, large objects are visible both in the display screen and by the naked eye, but in an inconsistent way because of the difference in perspective parameters between the device's camera and the user's eye [27]. Again, employing additional sensors and adjusting the camera focus [28] can alleviate the problem, but it is practically not feasible for the handheld platforms which strive to be stand-alone, self contained, and inexpensive.…”

User-Perspective Rendering for Handheld Applications

“…The external calibration of a mobile device aims at finding the relative pose between the front camera and the screen, which is rather important since all the following geometric poses are calculated based on it. However, this step has not been well studied in some previous user-perspective rendering algorithms [1,3,4,6,14,16,18,22,23]. They either use specifically designed systems of which the screencamera poses cannot be generalized out of their project contexts [1,3,14,16,18,22,23], or the poses are manually measured for their mobile devices [4,6].…”

“…Plus, we add a primary user face tracker to deal with the drawback that the classifier always tries to detect all faces appearing in the image while only one of them belongs to the primary user. Like in the works [2,16,22], we also track the user's face using RGB front camera.…”

“…There exist many works on user-perspective rendering [2,3,6,11,16,[18][19][20][21] for AR or entertainment applications. All of them only focus on the visualization which renders correct geometric shapes Figure 1: Top view of a smartphone facing a mirror with an arbitrary angle.…”