Developing a Novel Hands-Free Interaction Technique Based on Nose and Teeth Movements for Using Mobile Devices

Abstract

Human-mobile interaction is aimed at facilitating interaction with the smartphone devices. The conventional way to interact with mobile devices is through manual input where most of the applications are made assuming that the end user has full control over their hand movements. However, this assumption excludes people who are unable to use their hands or have suffered limb damage. In this paper, we proposed a nose and teeth based interaction system, which allows the users to control their mobile devices completely hands free. The proposed system uses the front facing camera of the smartphone to track the position of the nose for cursor control on the smartphone screen. The system detects teeth for performing the touch screen events such as tap, scroll, long press, and drag. Viola-Jones algorithm is used to detect the face and teeth based on the Haar features. After detecting the face, the nose position is calculated and tracked continuously using Lucas Kanade's method for optical flow estimation. All the touch screen events have been implemented in the system so that the user can execute all the operations of the smartphone. To evaluate the performance and the effect of (smartphone) device type on the execution time, the proposed system was installed in 3 smartphone devices and 7 trials for each device were performed by 3 different able-bodied elderly persons. The result shows a significant success rate for the detection of nose and teeth, and for the execution of the operations. The execution time of each operation slightly varies by 0.72s on average because of the configuration of the smartphones.INDEX TERMS HCI, human-mobile interaction, gesture operations, disabled user, accessibility, mobile device, smartphone.

Head Tracking and Voice Recognition Using Open Cv in Python

“…After that, the user can switch to the joystick mode to more precisely control where the pointer is located. If a non-rigid alteration is detected in the contour of the face, then any of the two modes can be activated or deactivated [19]. It uses standard infrared (IR) sensors to track head movements and a standard microcontroller (MCU) to map the sensor data to either relative movement (joystick mode) or an exact location on the screen (direct map-ping mode).…”

“…It uses standard infrared (IR) sensors to track head movements and a standard microcontroller (MCU) to map the sensor data to either relative movement (joystick mode) or an exact location on the screen (direct map-ping mode). Even though the way it works is pretty simple in theory, the system has to deal with several problems, such as optical noise from different sources, how people move their heads, accuracy, and power consumption [19].…”

A deep learning based multimodal interaction system for bed ridden and immobile hospital admitted patients: design, development and evaluation

“…This system not only showed how nose and teeth movement can be interpreted for alternate means of interaction but also demonstrated the importance of system evaluation for the justification of such systems. Islam et al [ 34 ] developed and evaluated another nose tracking cursor control system but it was developed for using Android smartphones. A disabled user unable to use his/her hands can use the system to perform all the basic touch operations and button operations.…”