Hand-free natural user interface for VR HMD with IR based facial gesture tracking sensor

Kim, Jinhyuk; Cha, Jaekwang; Lee, Hojun; Kim, Shiho

doi:10.1145/3139131.3143420

Cited by 13 publications

(7 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By using these methods, it is possible to determine the user's eye gaze (i.e., where the user is looking at in the VE), enabling them to point or aim at virtual elements (e.g., buttons) without moving the head [8]. However, it is also possible to determine relative eye movements (e.g., the eyes turned to the right) and issue a "to right" command [108], and detect when users blink [62,113] or have their eyes closed [59] to confirm selections or control the VR systems. Additionally, using the eyes for interaction suffers from what is described as the "Midas Touch" problem [50].…”

Section: Interaction Interfacesmentioning

confidence: 99%

Hands-free interaction in immersive virtual reality: A systematic review

Monteiro

Gonçalves

Coelho

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Hands are the most important tool to interact with virtual environments, and they should be available to perform the most critical tasks. For example, a surgeon in VR should keep his/her hands on the instruments and be able to do secondary tasks without performing a disruptive event to the operative task. In this common scenario, one can observe that hands are not available for interaction. The goal of this systematic review is to survey the literature and identify which hands-free interfaces are used, the performed interaction tasks, what metrics are used for interface evaluation, and the results of such evaluations. From 80 studies that met the eligibility criteria, the voice is the most studied interface, followed by the eye and head gaze. Some novel interfaces were brain interfaces and face expressions. System control and selection represent most of the interaction tasks studied and most studies evaluate interfaces for usability. Despite the best interface depending on the task and study, the voice was found to be versatile and showed good results amongst the studies. More research is recommended to improve the practical use of the interfaces and to evaluate the interfaces more formally.

show abstract

Section: Interaction Interfacesmentioning

confidence: 99%

Hands-free interaction in immersive virtual reality: A systematic review

Monteiro

Gonçalves

Coelho

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…Nickel et al found that “Langer ’s line” [32], which indicates the collagen fiber distribution in human skin, follows the direction of measured SRDR of skin [4], and Cha et al showed that the SRDR measured in relation to skin deformation mirrors deformation of the skin [5]. Cha et al and Kim et al applied IR diffusion reflectance to detect VR headset user’s facial expressions [33,34].…”

Section: Sensor Implementationmentioning

confidence: 99%

Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning

Cha

Kim

2019

Sensors

Self Cite

View full text Add to dashboard Cite

Developing a user interface (UI) suitable for headset environments is one of the challenges in the field of augmented reality (AR) technologies. This study proposes a hands-free UI for an AR headset that exploits facial gestures of the wearer to recognize user intentions. The facial gestures of the headset wearer are detected by a custom-designed sensor that detects skin deformation based on infrared diffusion characteristics of human skin. We designed a deep neural network classifier to determine the user’s intended gestures from skin-deformation data, which are exploited as user input commands for the proposed UI system. The proposed classifier is composed of a spatiotemporal autoencoder and deep embedded clustering algorithm, trained in an unsupervised manner. The UI device was embedded in a commercial AR headset, and several experiments were performed on the online sensor data to verify operation of the device. We achieved implementation of a hands-free UI for an AR headset with average accuracy of 95.4% user-command recognition, as determined through tests by participants.

show abstract

“…Especially, because AR/VR displays are placed near to eyes, it is impossible to touch the screen directly. Therefore, other input tools using various sensors such as leap motion sensors, electromyograph sensors, inertial measurement units, eye-trackers, IR facial gesture sensors, cameras, and axis-tilt sensors, have been employed [126][127][128][129][130][131][132][133][134].…”

Section: Introductionmentioning

confidence: 99%

Review of Capacitive Touchscreen Technologies: Overview, Research Trends, and Machine Learning Approaches

Nam

Seol

Lee

et al. 2021

Sensors

View full text Add to dashboard Cite

Touchscreens have been studied and developed for a long time to provide user-friendly and intuitive interfaces on displays. This paper describes the touchscreen technologies in four categories of resistive, capacitive, acoustic wave, and optical methods. Then, it addresses the main studies of SNR improvement and stylus support on the capacitive touchscreens that have been widely adopted in most consumer electronics such as smartphones, tablet PCs, and notebook PCs. In addition, the machine learning approaches for capacitive touchscreens are explained in four applications of user identification/authentication, gesture detection, accuracy improvement, and input discrimination.

show abstract

Hand-free natural user interface for VR HMD with IR based facial gesture tracking sensor

Cited by 13 publications

References 3 publications

Hands-free interaction in immersive virtual reality: A systematic review

Hands-free interaction in immersive virtual reality: A systematic review

Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning

Review of Capacitive Touchscreen Technologies: Overview, Research Trends, and Machine Learning Approaches

Contact Info

Product

Resources

About