Adaptive camera-based color mapping for mixed-reality applications

Abstract

We present a novel adaptive color mapping method for virtual objects in mixed-reality environments. In several mixed-reality applications, added virtual objects should be visually indistinguishable from real objects. Recent mixed-reality methods use globalillumination algorithms to approach this goal. However, simulating the light distribution is not enough for visually plausible images. Since the observing camera has its very own transfer function from real-world radiance values to RGB colors, virtual objects look artificial just because their rendered colors do not match with those of the camera. Our approach combines an on-line camera characterization method with a heuristic to map colors of virtual objects to colors as they would be seen by the observing camera. Previous tonemapping functions were not designed for use in mixed-reality systems and thus did not take the camera-specific behavior into account. In contrast, our method takes the camera into account and thus can also handle changes of its parameters during runtime. The results show that virtual objects look visually more plausible than by just applying tone-mapping operators.

Photorealism in Mixed Reality: A Systematic Literature Review

“…Zhang et al (2019) performs a 3D reconstruction using multiple RGB images as input. For the works from Knecht et al (2011); Grosch et al (2007); Kan and Kaufmann (2012); Rohmer et al (2014) and Tuceryan et al (2019), the 3D geometry of the scene was previously known.…”

“…To identify lighting conditions of the real scene, such as the position of the light source, many works, Knecht et al (2011);Grosch et al (2007); Kan and Kaufmann (2013); Franke (2013); Kan and Kaufmann (2012); Rohmer et al (2014) and Gierlinger et al (2010), use cameras with fish-eye lens. In Schwandt and Broll (2016) and Pereira et al (2020) however, the position of the light source was previously known, and in Marques et al (2018) A perceived trend was the use of Differential Rendering by Debevec (2008) as in Kan and Kaufmann (2013); Rohmer et al (2017); Franke (2013); Kan and Kaufmann (2012); Gruber et al (2015) and Gruber et al (2012).…”

“…In Schwandt and Broll (2016) and Pereira et al (2020) however, the position of the light source was previously known, and in Marques et al (2018) A perceived trend was the use of Differential Rendering by Debevec (2008) as in Kan and Kaufmann (2013); Rohmer et al (2017); Franke (2013); Kan and Kaufmann (2012); Gruber et al (2015) and Gruber et al (2012). In Knecht et al (2011) and Knecht et al (2012) a specific version, the Differential Instant Radiosity was used (Knecht et al, 2010). Instant Radiosity by Keller (1997) alone was used in Park et al (2016) and another technique for rendering was used in Gierlinger et al (2010), the Precomputed Radiance Transfer (Sloan et al, 2002).…”

“…Knecht et al (2012) presented a distributed illumination approach for AR with consistent illumination of virtual objects with direct light, indirect light, and shadows of primary and strong secondary lights. The paper from Knecht et al (2011) presents a solution for the photorealistic rendering of synthetic objects into real dynamic scenes in Augmented Reality applications. The work presented in Rohmer et al (2017) proposed a method for visually coherent interactive AR for mobile devices.…”

Design and evaluation of a low-cost 3D interaction technique for wearable and handled AR devices

“…Regarding the determination of the position and orientation of the hand or the gesture recognition, some works are based on the hardware aspect using a Kinect [11,12], or a flystick [13,14], or other materials, which gives accurate results, but with height cost. Other ones are vision based, using different vision algorithms [8,10].…”