User interaction in hands-free gaming: a comparative study of gaze-voice and touchscreen interface control

Uludağli, Muhtar Çağkan; Acartürk, Cengiz

doi:10.3906/elk-1710-128

Cited by 16 publications

(5 citation statements)

References 21 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since has a capacity to carry rich information from a human to a machine, making it to play such an oversimplified role can be perceived as counterintuitive and boring, especially to healthy users. This could be one of the reasons why in all previous attempts to combine voice and gaze in a hybrid human-machine interface, more complex voice commands were explored [12][13][14][15][16][17][18][19][20]. However, the participants of our study successfully fulfilled multiple selections on a computer screen using the simple "voice-as-click" approach, only slightly slower than with mouse-click confirmation.…”

Section: Discussionmentioning

confidence: 89%

“…The gaze is often used together with a voice in natural communication, e.g., in joint attention [9], in turn-taking [10], and in direct gaze interaction [11]. Attempts were made to combine gaze-and voice-based interaction in a single technology [12][13][14][15][16][17][18][19][20]. Indeed, it seems natural to use voice communication as a most natural and information-rich channel, and to supplement it with gaze pointing whenever spatial information is helpful, e.g., to quickly indicate where or on which object an action should be applied [21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection

et al. 2020

View full text Add to dashboard Cite

Voice- and gaze-based hands-free input are increasingly used in human-machine interaction. Attempts to combine them into a hybrid technology typically employ the voice channel as an information-rich channel. Voice seems to be “overqualified” to serve simply as a substitute of a computer mouse click, to confirm selections made by gaze. It could be expected that the user would feel discomfort if they had to frequently make “clicks” using their voice, or easily get bored, which also could lead to low performance. To test this, we asked 23 healthy participants to select moving objects with smooth pursuit eye movements. Manual confirmation of selection was faster and rated as more convenient than voice-based confirmation. However, the difference was not high, especially when voice was used to pronounce objects’ numbers (speech recognition was not applied): Score of convenience (M ± SD) was 9.2 ± 1.1 for manual and 8.0 ± 2.1 for voice confirmation, and time spent per object was 1269 ± 265 ms and 1626 ± 331 ms, respectively. We conclude that “voice-as-click” can be used to confirm selection in gaze-based interaction with computers as a substitute for the computer mouse click when manual confirmation cannot be used.

show abstract

Section: Discussionmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Despite their worse game performance in gaze-voice control, their participants reported a higher level of immersion for gaze-voice control compared to keyboard-mouse control. Uludagli and Acarturk [37] conducted a comparative study of gaze-voice and touchscreen interface control. They found that the participants exhibited higher game performance when they used the touchscreen control and that participants who used the gaze-voice control method were more psychologically absorbed.…”

Section: Eye Gaze and Voice In Computer Gamesmentioning

confidence: 99%

All Birds Must Fly: The Experience of Multimodal Hands-free Gaming with Gaze and Nonverbal Voice Synchronization

Hedeshy

Kumar

Lauer

et al. 2022

Proceedings of the 2022 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Eye tracking has evolved as a promising hands-free interaction mechanism to support people with disabilities. However, its adoption as a control mechanism in the gaming environment is constrained due to erroneous recognition of user intention and commands. Previous studies have suggested combining eye gaze with other modalities like voice input for improved interaction experience. However, speech recognition latency and accuracy is a major bottleneck, and the use of dictated verbal commands can disrupt the flow in gaming environment. Furthermore, several people with physical disabilities also suffer from speech impairments to utter precise verbal voice commands. In this work, we introduce nonverbal voice interaction (NVVI) to synchronize with gaze for an intuitive hands-free gaming experience. We propose gaze and NVVI (e.g., humming) for a spatio-temporal interaction applicable to several modern gaming apps, and developed 'All Birds Must Fly' as a representative app. In the experiment, we first compared the gameplay experience of gaze and NVVI (GV) with the conventional mouse and keyboard (MK) in a study with 15 non-disabled participants. The participants could effectively control the game environment with GV (expectedly a bit slower than MK). More importantly, they found GV more engaging, fun, and enjoyable. In a second study with 10 participants, we successfully validated the feasibility of GV with a target user group of people with disabilities.

show abstract

“…Similarly, van der Kemp and Sundstedt proposed a drawing system controlled by voice and gaze, opening new possibilities for creative expression and giving the wider user population access to this kind of application [57]. Further, researchers have combined gaze and voice-based interactions to enhance the immersive experience of players in games [43,56]. In line with previous work, we recognise the benefit of combining these modalities and present a voice note-taking system which combines the natural behaviour of gaze while reading with the convenience and expressiveness of voice for annotating text.…”

Section: Gaze and Voice-based Interactionmentioning

confidence: 99%

Gavin

Khan

Newn

Kelly

et al. 2021

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

Annotation is an effective reading strategy people often undertake while interacting with digital text. It involves highlighting pieces of text and making notes about them. Annotating while reading in a desktop environment is considered trivial but, in a mobile setting where people read while hand-holding devices, the task of highlighting and typing notes on a mobile display is challenging. In this article, we introduce GAVIN, a gaze-assisted voice note-taking application, which enables readers to seamlessly take voice notes on digital documents by implicitly anchoring them to text passages. We first conducted a contextual enquiry focusing on participants’ note-taking practices on digital documents. Using these findings, we propose a method which leverages eye-tracking and machine learning techniques to annotate voice notes with reference text passages. To evaluate our approach, we recruited 32 participants performing voice note-taking. Following, we trained a classifier on the data collected to predict text passage where participants made voice notes. Lastly, we employed the classifier to built GAVIN and conducted a user study to demonstrate the feasibility of the system. This research demonstrates the feasibility of using gaze as a resource for implicit anchoring of voice notes, enabling the design of systems that allow users to record voice notes with minimal effort and high accuracy.

show abstract

User interaction in hands-free gaming: a comparative study of gaze-voice and touchscreen interface control

Cited by 16 publications

References 21 publications

Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection

Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection

All Birds Must Fly: The Experience of Multimodal Hands-free Gaming with Gaze and Nonverbal Voice Synchronization

Gavin

Contact Info

Product

Resources

About