Enhanced Self-Perception in Mixed Reality: Egocentric Arm Segmentation and Database with Automatic Labelling

Abstract

In this study, we focus on the egocentric segmentation of arms to improve self-perception in Augmented Virtuality (AV). The main contributions of this work are: i) a comprehensive survey of segmentation algorithms for AV; ii) an Egocentric Arm Segmentation Dataset, composed of more than 10, 000 images, comprising variations of skin color, and gender, among others. We provide all details required for the automated generation of groundtruth and semi-synthetic images; iii) the use of deep learning for the first time for segmenting arms in AV; iv) to showcase the usefulness of this database, we report results on different real egocentric hand datasets, including GTEA Gaze+, EDSH, EgoHands, Ego Youtube Hands, THU-Read, TEgO, FPAB, and Ego Gesture, which allow for direct comparisons with existing approaches utilizing color or depth. Results confirm the suitability of the EgoArm dataset for this task, achieving improvement up to 40% with respect to the original network, depending on the particular dataset. Results also suggest that, while approaches based on color or depth can work in controlled conditions (lack of occlusion, uniform lighting, only objects of interest in the near range, controlled background, etc.), egocentric segmentation based on deep learning is more robust in real AV applications.

Egocentric Human Segmentation for Mixed Reality

“…Pigny et Dominjon [11] were the first to propose a deep algorithm to segment egocentric bodies. Their architecture was based on U-NET and trained using a hybrid dataset composed of images from the COCO dataset belonging to persons and a 1500-image custom dataset created following the automatic labelling procedure reported in [6]. They reported 16 ms of inference time for 256 × 256 images, and also observed problems of false positives that downgrade the AV experience.…”

“…They reported 16 ms of inference time for 256 × 256 images, and also observed problems of false positives that downgrade the AV experience. In this work we plan to extend our previous work [6] and explore further this segmentation problem targeting to perform real-time segmentation at higher resolutions, which are needed for creating and achieving a realistic immersive experience.…”

“…Despite effective for some situations, they still lack enough precision to provide a generic, realistic and real time immersive experience. In our previous work [6], we explore deep semantic segmentation networks to perform arm segmentation using the EgoArm semi-synthetic dataset. Results proved their increased robustness against uncontrolled scenarios with respect to color-based or depth-based approaches, but the particular architecture explored was not light enough for realtime performance.…”

Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks

“…The ability to localize human body parts from images is an essential requirement for many robotics and perception applications such as human-robot interaction [14], surgical robotics and medical images analysis [1], self-perception in mixed reality [9], and 3D human body reconstruction [25]. A collaborative robot, for example, should be able to locate the arms of the human operator with whom it is collaborating, just as a surgical robot must be able to accurately locate the limb on which it is operating.…”

“…Background removal systems can be used for body part localization as well: unfortunately, the generality of those systems allows to obtain acceptable results only in specific cases where the foreground contains only human limbs, or when a rich prior knowledge about the scene is given [18]. Other techniques, such as background removal in a green screen setting [9], allow to speed up the annotation process. However, they may come at the cost of the unrealistic feature of green color bleeding at the annotation boundaries.…”