Using audio cues to support motion gesture interaction on mobile devices

Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

2017

This paper presents an investigation into the effects of different feedback modalities on mid-air gesture interaction for infotainment systems in cars. Car crashes and near-crash events are most commonly caused by driver distraction. Mid-air interaction is a way of reducing driver distraction by reducing visual demand from infotainment. Despite a range of available modalities, feedback in mid-air gesture systems is generally provided through visual displays. We conducted a simulated driving study to investigate how different types of multimodal feedback can support in-air gestures. The effects of different feedback modalities on eye gaze behaviour, and the driving and gesturing tasks are considered. We found that feedback modality influenced gesturing behaviour. However, drivers corrected falsely executed gestures more often in non-visual conditions. Our findings show that non-visual feedback can reduce visual distraction significantly.

Section: Introductionmentioning

confidence: 99%

Novel Multimodal Feedback Techniques for In-Car Mid-Air Gesture Interaction

Shakeri

Williamson

Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

2017

“…It reduces mental workload [39,32] and provides a sense of control of the outcomes of one's actions [8]. It decreases eyes-off-the-road time [41] and improves user satisfaction [33]. For these reasons, a growing body of research is investigating feedback for mid-air interaction in driving situations.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, the driver's unfamiliarity with a gesture system has a negative impact on lane keeping ability and mental workload [19,38]. The decoupling of the user's hand from the interface [7] and the lack of sense of control over the touch less interface [8] are additional aspects which lessen user satisfaction, safety and usability of gestures [33].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

May the Force Be with You

Shakeri

Williamson

Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

2018

The use of ultrasound haptic feedback for mid-air gestures in cars has been proposed to provide a sense of control over the user's intended actions and to add touch to a touchless interaction. However, the impact of ultrasound feedback to the gesturing hand regarding lane deviation, eyes-off-the-road time (EORT) and perceived mental demand has not yet been measured. This paper investigates the impact of uni-and multimodal presentation of ultrasound feedback on the primary driving task and the secondary gesturing task in a simulated driving environment. The multimodal combinations of ultrasound included visual, auditory, and peripheral lights. We found that ultrasound feedback presented uni-modally and bi-modally resulted in significantly less EORT compared to visual feedback. Our results suggest that multimodal ultrasound feedback for mid-air interaction decreases EORT whilst not compromising driving performance nor mental demand and thus can increase safety while driving. CCS Concepts •Human-centered computing → Haptic devices; Auditory feedback; Gestural input;

“…Others have investigated how audio displays can be used by gesture systems. For example: Morrison et al [33] used sound to tell users which of their body parts a gesture-sensing system could and could not "see"; Charbonneau et al [11] used sound to help teach body gestures in games; and Morrison-Smith and Ruiz [34] used audio feedback to teach motion gestures.…”

Section: Giving Gesture Feedbackmentioning

confidence: 99%

Do That, There

Freeman

Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Lantz

2016

When users want to interact with an in-air gesture system, they must first address it. This involves finding where to gesture so that their actions can be sensed, and how to direct their input towards that system so that they do not also affect others or cause unwanted effects. This is an important problem [6] which lacks a practical solution. We present an interaction technique which uses multimodal feedback to help users address in-air gesture systems. The feedback tells them how ("do that") and where ("there") to gesture, using light, audio and tactile displays. By doing that there, users can direct their input to the system they wish to interact with, in a place where their gestures can be sensed. We discuss the design of our technique and three experiments investigating its use, finding that users can "do that" well (93.2%-99.9%) while accurately (51mm-80mm) and quickly (3.7s) finding "there".