Low-Cost Telepresence for Collaborative Virtual Environments

Abstract

Abstract-We present a novel low-cost method for visual communication and telepresence in a CAVE TM -like environment, relying on 2D stereo-based video avatars. The system combines a selection of proven efficient algorithms and approximations in a unique way, resulting in a convincing stereoscopic real-time representation of a remote user acquired in a spatially immersive display. The system was designed to extend existing projection systems with acquisition capabilities requiring minimal hardware modifications and cost. The system uses infrared-based image segmentation to enable concurrent acquisition and projection in an immersive environment without a static background. The system consists of two color cameras and two additional b/w cameras used for segmentation in the near-IR spectrum. There is no need for special optics as the mask and color image are merged using image-warping based on a depth estimation. The resulting stereo image stream is compressed, streamed across a network, and displayed as a frame-sequential stereo texture on a billboard in the remote virtual environment.

Correcting User's Head and Body Orientation Using a Comfort Pose Function

“…One of the main reasons for the delay in the widespread use of VR is the need for reducing the cost and complexity of the devices necessary to track the body and to provide immersive visualization. A number of previous works propose some kind of low cost techniques for tracking [3]. Despite the lower cost, however, many of those systems were still based on complex infrastructure to detect three-dimensional data.…”

Stereoscopic 3D copy & paste

“…Finally the copied object(s) is pasted into the target at some desired location, and the result is a composited stereo pair of images. Rhee et al [2007] introduced the concept of stereo billboards as planar proxies for stereoscopic telepresence display under the assumption that objects are always humans and fronto-parallel to the camera. In contrast, our stereo billboards are more general: they can represent arbitrary 3D objects, and the (optimal) orientations are computed using the objects' reconstructed 3D points as constraints.…”

Feasibility of the living canvas

“…In particular, the system builds and improves upon the lessons learned during the implementation of the blue-c tele-presence system [1] and a successive low-cost version based on two pairs of colour/IR cameras for stereoscopic user acquisition [2]. The acquisition problem has also been solved in a similar fashion for the construction of the Lightstage [12], where the background is lit uniformly to reveal the silhouette in the near IR spectrum, while a colour camera acquires the foreground image.…”

“…Initial experiments were conducted successfully with frontal IR lighting against both a dark background and a back-projection screen on the small-scale stage, similar in concept to [2]. With both backgrounds, the garments are significantly more reflective than the stage set.…”

“…The project builds upon previous work to acquire images of people standing inside a spatially immersive display used for virtual reality applications [1,2]. It reverses the problem in that it does not aim to acquire the geometry or texture of the performer, but instead project onto them.…”