Voice as a Mouse Click: Usability and Effectiveness of Simplified Hands-Free Gaze-Voice Selection

Abstract

Voice- and gaze-based hands-free input are increasingly used in human-machine interaction. Attempts to combine them into a hybrid technology typically employ the voice channel as an information-rich channel. Voice seems to be “overqualified” to serve simply as a substitute of a computer mouse click, to confirm selections made by gaze. It could be expected that the user would feel discomfort if they had to frequently make “clicks” using their voice, or easily get bored, which also could lead to low performance. To test this, we asked 23 healthy participants to select moving objects with smooth pursuit eye movements. Manual confirmation of selection was faster and rated as more convenient than voice-based confirmation. However, the difference was not high, especially when voice was used to pronounce objects’ numbers (speech recognition was not applied): Score of convenience (M ± SD) was 9.2 ± 1.1 for manual and 8.0 ± 2.1 for voice confirmation, and time spent per object was 1269 ± 265 ms and 1626 ± 331 ms, respectively. We conclude that “voice-as-click” can be used to confirm selection in gaze-based interaction with computers as a substitute for the computer mouse click when manual confirmation cannot be used.

Examining the impact of head‐mounted augmented reality on learning engineering drawings: A case study for three‐view drawing

“…Gaze interaction enables a natural gaze at the object of attention. Moreover, gaze and voice work more effectively together, which does not distract the user spatially, as voice and gaze may be directed at the same information recipient, suitable for quickly indicating where or on which object an action should be applied [75]. These three interaction modalities work together since students are more likely to develop meaningful learning in environments with multimodal interaction [50], contributing to heightened user enthusiasm [53].…”

“…In daily communication, gaze is often used in conjunction with voice. However, this combined technique has not been widely used in systems to date [75]. To quickly and accurately select virtual objects and perform the corresponding actions, a combination of gaze and voice interaction is applied.…”

Extended Control With Hybrid Gaze-BCI for Multi-Robot System Under Hands-Occupied Dual-Tasking

“…The eye movement signals have also been capitalized on to realize hands-free intuitive interfaces for selecting the moving targets [7], [8], [9] or still targets [15]. In particular, our study concentrates on selecting still targets, and the development of such gazed-based interface suffers from Midas Touch problem [10], i.e., not every location we fixate is related to the target.…”

“…For this reason, the dual-task paradigm has been widely adopted [29], [30], [31] to simulate the highly cognitive-demanding operation scenarios in real world, where the performance measurements of the primary task reflect the operator's manipulation ability under high cognitive load, and those of a secondary task provide objective information about the operator's cognitive load. Most previous studies have investigated the BCI-based and gaze-based control performance in the singletask paradigm [7], [8], [9], [32], [33], [34], [35], whereas few efforts have been made in a more demanding dual-task paradigm where the operator's both hands are involved until very recently.…”

“…Harnessing the gaze to infer the operator's intended onscreen commands promises to be a natural control interface for human-MRS interactions [7], [8], [9], [10]. Nevertheless, accuracy of gaze-based input is found to largely depend on the target size [11].…”