Novel Multimodal Feedback Techniques for In-Car Mid-Air Gesture Interaction

Abstract

This paper presents an investigation into the effects of different feedback modalities on mid-air gesture interaction for infotainment systems in cars. Car crashes and near-crash events are most commonly caused by driver distraction. Mid-air interaction is a way of reducing driver distraction by reducing visual demand from infotainment. Despite a range of available modalities, feedback in mid-air gesture systems is generally provided through visual displays. We conducted a simulated driving study to investigate how different types of multimodal feedback can support in-air gestures. The effects of different feedback modalities on eye gaze behaviour, and the driving and gesturing tasks are considered. We found that feedback modality influenced gesturing behaviour. However, drivers corrected falsely executed gestures more often in non-visual conditions. Our findings show that non-visual feedback can reduce visual distraction significantly.

May the Force Be with You

“…It reduces mental workload [39,32] and provides a sense of control of the outcomes of one's actions [8]. It decreases eyes-off-the-road time [41] and improves user satisfaction [33]. For these reasons, a growing body of research is investigating feedback for mid-air interaction in driving situations.…”

“…Shakeri et al [41] have investigated different unimodal feedback types such as visual, auditory, peripheral visual, and cutaneous push haptic feedback for mid-air gesture interaction in cars. They found that non-visual feedback reduced eyesoff-the-road time significantly and did not influence driving performance negatively.…”

Bimodal feedback for in-car mid-air gesture interaction

“…In a driving situation, traditional visual feedback is located in an eyes-off-the-road position whereas peripheral feedback can be perceived whilst looking ahead. Peripheral feedback has been shown to decrease eyes-off-the-road time compared to traditional visual feedback for mid-air gesture interaction [8]. Cutaneous push feedback on the steering wheel can be presented to the steering hand when gesturing.…”

“…In a previous study on unimodal feedback for mid-air gestures [8], we investigated the effectiveness of visual, auditory, cutaneous push, and peripheral lights feedback for mid-air gestures in cars (see Figure 1). We found that providing non-visual feedback decreased eyes-off-the-road time significantly, however mental demand was higher in the non-visual conditions.…”

Movement Characteristics and Effects of GUI Design on How Older Adults Swipe in Mid-Air

“…Motion-based interaction through mid-air gestures has become increasingly popular with the advancement of motion sensors (e.g. Microsoft HoloLens, Leap Motion) and has been employed in a diverse range of applications such as interactive TV, vehicles, and public displays [9,10,13]. Research [4,11] has found that older adults (aged 60 and older) may face greater challenges when interacting through mid-air gestures due to age-related decline in motor control, limited range of motion [5] and a lack of familiarity with this novel input method [6,8,12], however this issue is far from being tackled and fully understood.…”