Interactive Near-Field Illumination for Photorealistic Augmented Reality with Varying Materials on Mobile Devices

Abstract

At present, photorealistic augmentation is not yet possible since the computational power of mobile devices is insufficient. Even streaming solutions from stationary PCs cause a latency that affects user interactions considerably. Therefore, we introduce a differential rendering method that allows for a consistent illumination of the inserted virtual objects on mobile devices, avoiding delays. The computation effort is shared between a stationary PC and the mobile devices to make use of the capacities available on both sides. The method is designed such that only a minimum amount of data has to be transferred asynchronously between the participants. This allows for an interactive illumination of virtual objects with a consistent appearance under both temporally and spatially varying real illumination conditions. To describe the complex near-field illumination in an indoor scenario, HDR video cameras are used to capture the illumination from multiple directions. In this way, sources of illumination can be considered that are not directly visible to the mobile device because of occlusions and the limited field of view. While our method focuses on Lambertian materials, we also provide some initial approaches to approximate non-diffuse virtual objects and thereby allow for a wider field of application at nearly the same cost.

An Empirical Evaluation of the Performance of Real-Time Illumination Approaches: Realistic Scenes in Augmented Reality

“…Their factorization of the light seeks to compute the dominant light direction and color separately using spherical harmonic (SH) coefficients which is obtained by a projection onto the SH basis functions. As these functions are orthonormal, the dot product was the amount of incident light from all directions at a certain vertex [16]. Gruber et al [6] also expressed the incident light field with SH to estimate the distance of the light field which represent every incident light ray on a hemisphere from many different observers of an arbitrary geometry.…”

“…Richter-Trummer et al [15] used inverse rendering which factored the texture color into incident light and the diffuse albedo color through the radiance transfer method. For more material estimation Rohmer et al [16] assumed a Lambertian environment for incident light integration over the surface of the upper hemisphere.…”

“…They sampled the cosine hemisphere for indirect illumination on the diffuse surfaces on the real and virtual objects and sampled the Phong lobe for glossy surfaces on the virtual objects only. Other sampling techniques were applied on the shadow mapping to create Virtual Point Lights (VPLs) which then enables the calculation of the indirect illumination [10,16].…”

Dynamic HDR environment capture for mixed reality

“…Placing a light probe at the virtual object location can be impractical or impossible in some situations, so other approaches focus on removing this requirement. Some approaches process information from light probes placed elsewhere in the scene to infer lighting at the virtual object location [11,37].…”

Synthesis of Environment Maps for Mixed Reality

“…Grosch et al [14] made use of this, placing a fisheye camera at the entrance to a room to measure incoming light, and then simulating light transport within the room to add virtual content. Rohmer et al [39] demonstrated a technique combining information from multiple fisheye cameras to collect detailed lighting over a volume. Other active light probes have also been developed, such as that of Matsuoka et al [34], who used a hemispherical array of photodiodes to find the dominant light direction.…”