Natural Environment Illumination: Coherent Interactive Augmented Reality for Mobile and Non-Mobile Devices

Abstract

Augmented Reality offers many applications today, especially on mobile devices. Due to the lack of mobile hardware for illumination measurements, photorealistic rendering with consistent appearance of virtual objects is still an area of active research. In this paper, we present a full two-stage pipeline for environment acquisition and augmentation of live camera images using a mobile device with a depth sensor. We show how to directly work on a recorded 3D point cloud of the real environment containing high dynamic range color values. For unknown and automatically changing camera settings, a color compensation method is introduced. Based on this, we show photorealistic augmentations using variants of differential light simulation techniques. The presented methods are tailored for mobile devices and run at interactive frame rates. However, our methods are scalable to trade performance for quality and can produce quality renderings on desktop hardware.

Interactive and Immersive Learning Using 360° Virtual Reality Contents on Mobile Platforms

“…e 360°contents, which are called 360°photographs or videos, are captured by several wide-angle cameras, which are stitched and mapped in a spherical or hemispherical space to observe the surrounding scenes based on the camera [2]. Most research works for 360°content creation focus on accurate stitching for spherical mapping [5,6], image-based lighting effects [7][8][9], resolution improvement, and distortion correction of images from wide-angle cameras [10]. Some researchers investigated methods to improve the interactivity in imagebased VR contents by adding URLs to 360°images [11], changing scenes using button clicks [12,13], and adding special effects to reduce the differences between composite boundaries [14].…”

Dynamic HDR environment capture for mixed reality

“…This then provides all the geometric and lighting information required to insert the virtual content. Zhang et al [54] showed how this kind of approach could be used in an MR application for virtually refurnishing rooms, and Rohmer et al [38] used a similar HDR model to render photorealistic virtual content on mobile and desktop hardware. These approaches capture rich scene information using just an RGBD sensor, but have the disadvantage of requiring the whole scene to be reconstructed before virtual content can be added.…”

“…Before projecting a new set of surfels corresponding to the current frame, we first convert the observed LDR colour values to HDR values in a linear colour space. We approximately linearise the input values by applying an inverse gamma correction (γ = 2.2), similarly to [38,54]. We then estimate the exposure and white balance applied by the RGBD colour camera.…”

“…We then estimate the exposure and white balance applied by the RGBD colour camera. As in [38], we model the exposure and white balance applied by the camera as a linear scaling applied to each colour channel.…”

“…During precomputation, we assume the virtual object will be placed on a Lambertian planar surface. Note, however, that the lighting information captured by our approach could be used as input to many other rendering approaches, such as those employed in [38].…”