Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

Abstract

In this paper, we present preliminary results on the use of deep learning techniques to integrate the user's selfbody and other participants into a head-mounted video seethrough augmented virtuality scenario. It has been previously shown that seeing user's bodies in such simulations may improve the feeling of both self and social presence in the virtual environment, as well as user performance. We propose to use a convolutional neural network for real time semantic segmentation of users' bodies in the stereoscopic RGB video streams acquired from the perspective of the user. We describe design issues as well as implementation details of the system and demonstrate the feasibility of using such neural networks for merging users' bodies in an augmented virtuality simulation.

Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks

“…Pioneering work [21] proposed the UNet network, an encoder-decoder architecture consisting of a contracting path to capture context and a symmetric expanding path that enables precise localization. Even though the network was developed for biomedical image segmentation, it generalized well to other domains, such as urban scenarios [27] and indoor environments [19]. Due to this, numerous variations of the UNet network have also been proposed.…”

Real Time Egocentric Segmentation for Video-self Avatar in Mixed Reality

“…In MR experiences, users usually see themselves in the form of a virtual graphical avatar. One alternative approach would be to use videobased self-avatars, by segmenting body limbs (arms, legs, or whole body) from the egocentric vision captured from a camera attached to a Virtual Reality (VR) device, as in Fig 1 . Previous approaches for bringing real bodies into MR have been based on: i) color information, allowing users to see their own hands/ bare arms [16]; ii) depth [14], by segmenting anything below a certain distance threshold or even deep learning to segment bare/clothes arms [6], or whole bodies [13]. However, those recent methods based on deep learning still fail at reaching sufficient execution speed.…”