Interactive chroma keying for mixed reality

Abstract

In Mixed Reality (MR) applications, immersion of virtual objects in captured video contributes to the perceived unification of two worlds, one real, one synthetic. Since virtual actors and surround may appear both closer and farther than real objects, compositing must consider spatial relationships in the resulting world. Chroma keying, often called blue screening or green screening, is one common solution to this problem. This method is under-constrained and most commonly addressed through a combination of environment preparation and commercial products. In interactive MR domains that impose restrictions on the video camera hardware, such as in experiences using video see-through (VST) head-mounted displays (HMD), chroma keying becomes even more difficult due to the relatively low camera quality, the use of multiple camera sources (one per eye), and the required processing speed. Dealing with these constraints requires a fast and affordable solution. In our approach, we precondition the chroma key by using principal component analysis (PCA) to obtain usable alpha mattes from video streams in real-time on commodity graphics processing units (GPUs). In addition, we demonstrate how our method compares to off-line commercial keying tools and how it performs with respect to signal noise within the video stream.

GPU‐based efficient realistic techniques for bleeding and smoke generation in surgical simulators

“…The technique that we have developed for smoke generation in virtual environments is based on the principle of capturing sample videos of smoke and overlaying them on the OpenGL frame buffer by removing the background42 (see Figure 1). In our study, the removal of the background was performed at the GPU (at the pixel shader), where the video frames are blended together to produce the final effect, as shown in Figure 1.…”

A Review on Mixed Reality: Current Trends, Challenges and Prospects

“…Ref. [50] proposed using chroma-keying with PCA for the immersion of virtual objects in video images. This method was useful for matting, which is a process used for foreground identification with separation images to the back and foreground.…”

“…Although the discussed methods have great benefits for close-to-reality visualizations in MR applications, there are important challenges impeding their use. In the chroma key, the use of statically-based techniques for temporal matting and Holoing artifacts is needed [50]. Improvement of the light and shadow models requires temporal and dynamic effects modeling (such as abrupt changes in lighting, dynamic scene, and local light source) in addition to specifying classes of different algorithms for different light paths [51,54].…”

Real-Time Video Matting for Mixed Reality Using Depth Generated Trimaps

“…This is commonly known as constant color matting, or chroma-keying. Solutions exist using color volumes (Bergh and Lalioti, 1999;Mishima, 1992;Beato et al, 2009) or specialized algorithms based on heuristics (Vlahos, 1978). While these methods run in real-time, they require training a model on the likely background color or handtuning parameters for optimal perception.…”

“…Global solutions such as (Levin et al, 2006;Levin et al, 2008;Shi and Malik, 2000) consider expensive processing of an nxn weight matrix, where n is the number of pixels in the image. Parallelizing a local minimizer to the matting linear system may be done via gradient descent or the conjugate gradient method (Gvili et al, 2003;Beato et al, 2009;Gastal and Oliveira, 2010). For these solutions to work well, the weight matrix must use global color assumptions.…”