Identification of the Driver's Interest Point using a Head Pose Trajectory for Situated Dialog Systems

Abstract

This paper addresses issues existing in situated language understanding in a moving car. Particularly, we propose a method for understanding user queries regarding specific target buildings in their surroundings based on the driver's head pose and speech information. To identify a meaningful head pose motion related to the user query that is among spontaneous motions while driving, we construct a model describing the relationship between sequences of a driver's head pose and the relative direction to an interest point using the Gaussian process regression. We also consider time-varying interest point using kernel density estimation. We collected situated queries from subject drivers by using our research system embedded in a real car. The proposed method achieves an improvement in the target identification rate by 14% in the user-independent training condition and 27% in the user-dependent training condition over the method that uses the head motion at the start-of-speech timing.

Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle

“…Moreover, referencing objects outside the vehicle has been investigated using different approaches and modalities. Rümelin et al [30] used free-hand pointing gestures, Fujimura et al [6] used hand-constrained pointing gestures, Kang et al [11] used eye gaze gestures, while Kim et al [13] and Misu et al [17] used speech-triggered head pose trajectories. However, these studies focused on single-modality approaches that were lacking in performance.…”

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

“…EyePointing [35] makes use of finger pointing to trigger the selection of objects on a screen using gaze direction. Misu et al [22] and Kim et al [17] make use of head pose using speech as a trigger for driver queries of outside-vehicle objects.…”

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

“…Therefore, researchers have attempted various approaches for controlling objects using symbolic hand gestures [22,38,45,52,54], deictic (pointing) hand gestures [23,25,34], eye gaze [13,24,27,31,36,37,50,51,53], and facial expressions [3,18,46]. Specifically for the automotive domain, in-vehicle interaction has been attempted using hand gestures [5,16,33,40], eye gaze [36], and facial expressions [44], while outside-the-vehicle interaction has been attempted using pointing gestures [17,41], eye gaze [24], and head pose [26,30]. Although most of the previous methods focus on single-modality approaches while using a button or voice commands as event triggers, more recent work focused on multimodal fusion approaches to enhance performance.…”