Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle

Gomaa, Amr; Reyes, Guillermo; Alles, Alexandra; Rupp, Lydia Helene; Feld, Michael S.

doi:10.1145/3382507.3418817

Cited by 16 publications

(26 citation statements)

References 33 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moniri et al [32] studied multimodal interaction in referencing objects outside the vehicle for passenger seat users using eye gaze, head pose, and pointing gestures and proposed an algorithm for the selection process. Similarly, Gomaa et al [19] investigated a multimodal referencing approach using gaze and pointing gestures while driving and highlighted personal differences in behavior among drivers. They proposed a rule-based approach for fusion that exploits the differences in users' behavior by modality switching between gaze and pointing based on a specific user or situation.…”

Section: Related Workmentioning

confidence: 99%

“…We designed a within-subject counterbalanced experiment using the same setup (as seen in Figure 2) as Gomaa et al [19] to collect data for the referencing task. We used OpenDS software [29] with a steering wheel, pedals, and speakers for the driving simulation; a state-of-the-art non-commercial hand tracking camera prototype specially designed for in-vehicle control for pointing tracking; and SMI Eye Tracking Glasses 1 for gaze tracking.…”

Section: Design and Participantsmentioning

confidence: 99%

“…We used OpenDS software [29] with a steering wheel, pedals, and speakers for the driving simulation; a state-of-the-art non-commercial hand tracking camera prototype specially designed for in-vehicle control for pointing tracking; and SMI Eye Tracking Glasses 1 for gaze tracking. For the driving task, we used the same 40-minute five-conditions star-shaped driving route with no traffic and a maximum driving speed of 60 km/hour as Gomaa et al [19]. The five counterbalanced conditions were divided into two levels of environment density (i.e., number of distractors around the Points of Interest (PoI)), two levels of distance from the road (i.e., near versus far PoI), and one autonomous driving condition.…”

Section: Design and Participantsmentioning

confidence: 99%

“…We consider a common 1D polar coordinate system with horizontal angular coordinates only (as seen in Figure 1) to which both pointing and gaze tracking systems as well as the PoI ground truth (GT) are transformed (similar to [19,24]). Since this is a continuous output case, we need to solve a regression problem.…”

Section: Fusion Approachmentioning

confidence: 99%

“…4.1.1 Late Fusion. The first approach involves pre-processing of the data, in which we transform the pointing and gaze raw data using the method in [19], then use these transformed horizontal angles as input features to our machine learning algorithm while using the PoI center angle as the target output (i.e., ground truth).…”

Section: Fusion Approachmentioning

confidence: 99%

See 4 more Smart Citations

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

Gomaa,

Reyes,

Feld

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Over the past decades, the addition of hundreds of sensors to modern vehicles has led to an exponential increase in their capabilities. This allows for novel approaches to interaction with the vehicle that go beyond traditional touch-based and voice command approaches, such as emotion recognition, head rotation, eye gaze, and pointing gestures. Although gaze and pointing gestures have been used before for referencing objects inside and outside vehicles, the multimodal interaction and fusion of these gestures have so far not been extensively studied. We propose a novel learning-based multimodal fusion approach for referencing outside-the-vehicle objects while maintaining a long driving route in a simulated environment. The proposed multimodal approaches outperform single-modality approaches in multiple aspects and conditions. Moreover, we also demonstrate possible ways to exploit behavioral differences between users when completing the referencing task to realize an adaptable personalized system for each driver. We propose a personalization technique based on the transfer-of-learning concept for exceedingly small data sizes to enhance prediction and adapt to individualistic referencing behavior. Our code is publicly available at https://github.com/amr-gomaa/ML-PersRef. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods; Pointing; Gestural input; • Computing methodologies → Support vector machines; Neural networks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Design and Participantsmentioning

confidence: 99%

Section: Design and Participantsmentioning

confidence: 99%

Section: Fusion Approachmentioning

confidence: 99%

Section: Fusion Approachmentioning

confidence: 99%

See 3 more Smart Citations

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

Gomaa,

Reyes,

Feld

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects

Aftab

Beeck

Rohrhirsch

et al. 2021

2021 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

Robust Multi-User In-Hand Object Recognition in Human-Robot Collaboration Using a Wearable Force-Myography Device

Bamani

Kahanowich

Ben-David

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In communication between humans, gestures are often preferred or complementary to verbal expression since the former offers better spatial referral. Finger pointing gesture conveys vital information regarding some point of interest in the environment. In human-robot interaction, a user can easily direct a robot to a target location, for example, in search and rescue or factory assistance. State-of-the-art approaches for visual pointing estimation often rely on depth cameras, are limited to indoor environments and provide discrete predictions between limited targets. In this paper, we explore the learning of models for robots to understand pointing directives in various indoor and outdoor environments solely based on a single RGB camera. A novel framework is proposed which includes a designated model termed PointingNet. PointingNet recognizes the occurrence of pointing followed by approximating the position and direction of the index finger. The model relies on a novel segmentation model for masking any lifted arm. While state-of-the-art human pose estimation models provide poor pointing angle estimation accuracy of 28 • , PointingNet exhibits mean accuracy of less than 2 • . With the pointing information, the target is computed followed by planning and motion of the robot. The framework is evaluated on two robotic systems yielding accurate target reaching.

show abstract

Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle

Cited by 16 publications

References 33 publications

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects

Robust Multi-User In-Hand Object Recognition in Human-Robot Collaboration Using a Wearable Force-Myography Device

Contact Info

Product

Resources

About