Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning

Cha, Jaekwang; Kim, Jinhyuk; Kim, Shiho

doi:10.3390/s19204441

Cited by 6 publications

(7 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… Various sensors can be attached to the body for motion capture, such as Smart Shoe, Pressure mat, sensors attached to leg, smartwatches, head sensors, data glove and Biometrics Goniometers and Torsiometers on toe and arm [ 15 , 16 , 17 , 18 , 19 ]. …”

Section: Figurementioning

confidence: 99%

See 1 more Smart Citation

Review of Wearable Devices and Data Collection Considerations for Connected Health

Vijayan

Connolly

Condell³

et al. 2021

Sensors

209

View full text Add to dashboard Cite

Wearable sensor technology has gradually extended its usability into a wide range of well-known applications. Wearable sensors can typically assess and quantify the wearer’s physiology and are commonly employed for human activity detection and quantified self-assessment. Wearable sensors are increasingly utilised to monitor patient health, rapidly assist with disease diagnosis, and help predict and often improve patient outcomes. Clinicians use various self-report questionnaires and well-known tests to report patient symptoms and assess their functional ability. These assessments are time consuming and costly and depend on subjective patient recall. Moreover, measurements may not accurately demonstrate the patient’s functional ability whilst at home. Wearable sensors can be used to detect and quantify specific movements in different applications. The volume of data collected by wearable sensors during long-term assessment of ambulatory movement can become immense in tuple size. This paper discusses current techniques used to track and record various human body movements, as well as techniques used to measure activity and sleep from long-term data collected by wearable technology devices.

show abstract

Section: Figurementioning

confidence: 99%

“…Sleep sensing may also be done using movement sensors. Athletes have several specific sensor-based applications, and Inertial Measurement Unit (IMU) sensors are now widely deployed in professional sports [ 15 ]. The cost of these applications is also usually too high for regular use in the health service.…”

Section: Introductionmentioning

confidence: 99%

Review of Wearable Devices and Data Collection Considerations for Connected Health

Vijayan

Connolly

Condell³

et al. 2021

Sensors

209

View full text Add to dashboard Cite

show abstract

“…Among those hands-free methods, the proposed interface utilizes facial skin movement as an input modality. The proposed user input device utilizing facial skin movements is a unique method for VR headsets [ 16 , 17 ].…”

Section: Introductionmentioning

confidence: 99%

“…The results indicate that the propagation of IR rays into tissue can be measured for various parts of the body. Furthermore, sensor modules with laser diodes and IR cameras instead of IR sensor pairs have been proposed [ 16 , 19 , 20 ]. The IR propagation characteristics can be used in AR environments in which the skin and sensor are not in close contact [ 16 ].…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, sensor modules with laser diodes and IR cameras instead of IR sensor pairs have been proposed [ 16 , 19 , 20 ]. The IR propagation characteristics can be used in AR environments in which the skin and sensor are not in close contact [ 16 ]. Using an unsupervised deep neural network for an AR headset, the classifier exhibited a user-command recognition rate of 95.4%.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hands-Free User Interface for VR Headsets Based on In Situ Facial Gesture Sensing

Kim

Cha

Kim

2020

Sensors

Self Cite

View full text Add to dashboard Cite

The typical configuration of virtual reality (VR) devices consists of a head-mounted display (HMD) and handheld controllers. As such, these units have limited utility in tasks that require hand-free operation, such as in surgical operations or assembly works in cyberspace. We propose a user interface for a VR headset based on a wearer’s facial gestures for hands-free interaction, similar to a touch interface. By sensing and recognizing the expressions associated with the in situ intentional movements of a user’s facial muscles, we define a set of commands that combine predefined facial gestures with head movements. This is achieved by utilizing six pairs of infrared (IR) photocouplers positioned at the foam interface of an HMD. We demonstrate the usability and report on the user experience as well as the performance of the proposed command set using an experimental VR game without any additional controllers. We obtained more than 99% of recognition accuracy for each facial gesture throughout the three steps of experimental tests. The proposed input interface is a cost-effective and efficient solution that facilitates hands-free user operation of a VR headset using built-in infrared photocouplers positioned in the foam interface. The proposed system recognizes facial gestures and incorporates a hands-free user interface to HMD, which is similar to the touch-screen experience of a smartphone.

show abstract

Deep Learning in Computational Design Synthesis: A Comprehensive Review

Kumar Singh,

Rai,

Pradip Khawale

et al. 2024

Journal of Computing and Information Science in Engineering

View full text Add to dashboard Cite

A paradigm shift in the computational design synthesis domain is being witnessed by the onset of the innovative usage of machine learning techniques. The rapidly evolving paradigmatic shift calls for systematic and comprehensive assimilation of extant knowledge at the intersection of machine learning and computational design synthesis. Understanding nuances, identifying research gaps, and outlining the future direction for cutting-edge research is imperative. This paper outlines a hybrid literature review consisting of a thematic and framework synthesis survey to enable conceptual synthesis of information at the convergence of computational design, machine learning, and big-data models. The thematic literature survey aims at conducting an in-depth descriptive survey along the lines of a broader theme of machine learning in computational design. The framework synthesis-based survey tries to encapsulate the research findings in a conceptual framework to understand the domain better. The framework is based on the computational design synthesis (CDS) process, which consists of four sub-modules: representation, generation, evaluation, and guidance. Each sub-module has undergone an analysis to identify potential research gaps and formulate research questions. Additionally, we consider the limitations of our study and pinpoint the realms where the research can be extended in the future.

show abstract

Hands-Free User Interface for AR/VR Devices Exploiting Wearer’s Facial Gestures Using Unsupervised Deep Learning

Cited by 6 publications

References 38 publications

Review of Wearable Devices and Data Collection Considerations for Connected Health

Review of Wearable Devices and Data Collection Considerations for Connected Health

Hands-Free User Interface for VR Headsets Based on In Situ Facial Gesture Sensing

Deep Learning in Computational Design Synthesis: A Comprehensive Review

Contact Info

Product

Resources

About