Enhanced Self-Perception in Mixed Reality: Egocentric Arm Segmentation and Database With Automatic Labeling

Abstract

In this study, we focus on the egocentric segmentation of arms to improve self-perception in Augmented Virtuality (AV). The main contributions of this work are: i) a comprehensive survey of segmentation algorithms for AV; ii) an Egocentric Arm Segmentation Dataset (EgoArm), composed of more than 10, 000 images, demographically inclusive (variations of skin color, and gender), and open for research purposes. We also provide all details required for the automated generation of groundtruth and semi-synthetic images; iii) the proposal of a deep learning network to segment arms in AV; iv) a detailed quantitative and qualitative evaluation to showcase the usefulness of the deep network and EgoArm dataset, reporting results on different real egocentric hand datasets, including GTEA Gaze+, EDSH, EgoHands, Ego Youtube Hands, THU-Read, TEgO, FPAB, and Ego Gesture, which allow for direct comparisons with existing approaches using color or depth. Results confirm the suitability of the EgoArm dataset for this task, achieving improvements up to 40% with respect to the baseline network, depending on the particular dataset. Results also suggest that, while approaches based on color or depth can work under controlled conditions (lack of occlusion, uniform lighting, only objects of interest in the near range, controlled background, etc.), deep learning is more robust in real AV applications.

Real Time Egocentric Object Segmentation: THU-READ Labeling and Benchmarking Results

“…From this AMT-based labeling experiment we found of special relevance that: i) better results were obtained from Turks who hold Master Qualification; and ii) instructors checked themselves that boundaries were precise enough and classes were corrected 5 . If images were not correctly processed, instructors rejected the tasks, providing detailed feedback.…”

“…To achieve this, one has to move to- wards deep learning approaches, and more in particular to semantic segmentation networks. In this line, Gonzalez-Sosa et al explore state-of-the-art networks for segmenting egocentric human body parts [5] using a semi-synthetic dataset. The reported results suggest suggested that semantic segmentation networks can overcome some of the limitations of color-based solutions, as now networks incorporate more complex information than just color, or of depth-based solutions, specially in outdoors scenarios.…”

“…Previous works have explored computer vision algorithms to integrate user's bodies into VR, either by using color-based approaches [1], depth-based approaches [8], and more recently deep-learning based ones [5]. One of the main benefits of deep learning approaches compared to color or depth solutions, is that apart from overcoming some of their limitations, they can jointly classify and segment objects of interest.…”

“…5 Done by overlapping labeled images created by Turks on top of their original RGB counterpart images…”

TCP-Based Distributed Offloading Architecture for the Future of Untethered Immersive Experiences in Wireless Networks

“…Fully immersive applications must provide the users with a high sense of embodiment and presence [21], which requires complex algorithms such as hand tracking or egocentric human segmentation [12]. The theoretical network requirements, in terms of latency and downlink and uplink throughput, for successfully ofoading some or all of these and other complex algorithms in diferent scenarios are proposed in [22].…”