Improvement of robustness against electrode shift for facial electromyogram-based facial expression recognition using domain adaptation in VR-based metaverse applications

Cha

et al. 2023

Sensors

Self Cite

With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user’s facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user’s face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users’ facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user’s facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user’s eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user’s facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications

Cha

et al. 2023

Sensors

Self Cite

AR‐Enabled Persistent Human–Machine Interfaces via a Scalable Soft Electrode Array

Kim,

Cha,

Kim

et al. 2023

Advanced Science

Augmented reality (AR) is a computer graphics technique that creates a seamless interface between the real and virtual worlds. AR usage rapidly spreads across diverse areas, such as healthcare, education, and entertainment. Despite its immense potential, AR interface controls rely on an external joystick, a smartphone, or a fixed camera system susceptible to lighting. Here, an AR‐integrated soft wearable electronic system that detects the gestures of a subject for more intuitive, accurate, and direct control of external systems is introduced. Specifically, a soft, all‐in‐one wearable device includes a scalable electrode array and integrated wireless system to measure electromyograms for real‐time continuous recognition of hand gestures. An advanced machine learning algorithm embedded in the system enables the classification of ten different classes with an accuracy of 96.08%. Compared to the conventional rigid wearables, the multi‐channel soft wearable system offers an enhanced signal‐to‐noise ratio and consistency over multiple uses due to skin conformality. The demonstration of the AR‐integrated soft wearable system for drone control captures the potential of the platform technology to offer numerous human–machine interface opportunities for users to interact remotely with external hardware and software.

Navigating the metaverse: unraveling the impact of artificial intelligence—a comprehensive review and gap analysis

Fadhel,

Duhaim,

Albahri

et al. 2024

Artif Intell Rev

In response to the burgeoning interest in the Metaverse—a virtual reality-driven immersive digital world—this study delves into the pivotal role of AI in shaping its functionalities and elevating user engagement. Focused on recent advancements, prevailing challenges, and potential future developments, our research draws from a comprehensive analysis grounded in meticulous methodology. The study, informed by credible sources including SD, Scopus, IEEE, and WoS, encompasses 846 retrieved studies. Through a rigorous selection process, 54 research papers were identified as relevant, forming the basis for a specific taxonomy of AI in the Metaverse. Our examination spans diverse dimensions of the Metaverse, encompassing augmented reality, virtual reality, mixed reality, Blockchain, Agent Systems, Intelligent NPCs, Societal and Educational Impact, HCI and Systems Design, and Technical Aspects. Emphasizing the necessity of adopting trustworthy AI in the Metaverse, our findings underscore its potential to enhance user experience, safeguard privacy, and promote responsible technology use. This paper not only sheds light on the scholarly interest in the Metaverse but also explores its impact on human behavior, education, societal norms, and community dynamics. Serving as a foundation for future development and responsible implementation of the Metaverse concept, our research identifies and addresses seven open issues, providing indispensable insights for subsequent studies on the integration of AI in the Metaverse.