Synthesis of Environment Maps for Mixed Reality

Abstract

When rendering virtual objects in a mixed reality application, it is helpful to have access to an environment map that captures the appearance of the scene from the perspective of the virtual object. It is straightforward to render virtual objects into such maps, but capturing and correctly rendering the real components of the scene into the map is much more challenging. This information is often recovered from physical light probes, such as reflective spheres or fisheye cameras, placed at the location of the virtual object in the scene. For many application areas, however, real light probes would be intrusive or impractical. Ideally, all of the information necessary to produce detailed environment maps could be captured using a single device. We introduce a method using an RGBD camera and a small fisheye camera, contained in a single unit, to create environment maps at any location in an indoor scene. The method combines the output from both cameras to correct for their limited field of view and the displacement from the virtual object, producing complete environment maps suitable for rendering the virtual content in real time. Our method improves on previous probeless approaches by its ability to recover high-frequency environment maps. We demonstrate how this can be used to render virtual objects which shadow, reflect and refract their environment convincingly.

A Classification of Augmented Reality Approaches for Spatial Data Visualization

“…the user needs to rotate the camera around to fully perceive the visualization. Such approaches in AR are often found to take place in a single room, as demonstrated in recent examples [36,41].…”

Beyond ‘mobility’: A new understanding of moving with technology

“…The most advanced and explicit stream of research investigating the mediating role of mobile technology is the emerging field of augmented reality, featuring devices such as Google Glass and Microsoft's Hololens and their associated applications (Walton, Thomas, Steed, & Sugimoto, 2017) such as in health (McNaney et al, 2014), gaming (Li & Fang, 2018), or sales (Xu, Liu, & Lee, 2018). This stream of research highlights how mobile technology acts as a medium that sits quite literally between the user and the environment and provides location-specific information overlayed directly onto (a representation of) the physical surroundings so that the user is essentially 'looking through' the device at the environment.…”

Remote Surgeon Hand Motion and Occlusion Removal in Mixed Reality in Breast Surgical Telepresence: Rural and Remote Care

“…This is useful for the proposed solution and the concept of alpha blending has been implemented. Walton et al (2017) has proposed a way to render virtual objects in real time in mixed reality in the multimedia field. It has used a RGBD and a small fisheye camera to create environment maps, which generate a 3D geometric model of the scene, using a dense 3D model from an RGBD camera using a Simultaneous Localization And Mapping (SLAM) algorithm.…”

“…Surgical guidance can be a significantly benefit MR Kersten-Oertel et al (2013) using implementation of mixed reality image-guided surgery with different visualization techniques based on a Data, Visualization processing, View (DVV) taxonomy (Walton et al, 2017), whereby 'a set of data about the patient is sent to the end user for image processing and visualizing. The result is a new defined perception location, a display technique to create the visualization and an interactive tool which deals with design.…”