ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

Gomaa, Amr; Reyes, Guillermo; Feld, Michael S.

doi:10.1145/3462244.3479910

Cited by 8 publications

(17 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, recent research advancement is oriented towards more user-centered design approaches for in-vehicle interfaces in order to alleviate the mental effort accompanying these added features [17,37,54,77,80]. Consequently, multiple designs emerged for seamless non-intrusive in-vehicle interfaces [1,10,12,32,36,39,40,55,57,94,99]. While these previous approaches and studies focus on enhancing user experience and reducing drivers' MWL through offline pre-design feedback (e.g., gathering users' requirements and designing a universal semi-customizable interface), others focus on real-time (and semi-real-time) approaches to obtain user feedback and adapt the interface based on the user's MWL and stress levels.…”

Section: Hmi Design In Automotive Domainmentioning

confidence: 99%

What’s on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving

Gomaa

Alles

Meiser

et al. 2022

Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Self Cite

View full text Add to dashboard Cite

Several researchers have focused on studying driver cognitive behavior and mental load for in-vehicle interaction while driving. Adaptive interfaces that vary with mental and perceptual load levels could help in reducing accidents and enhancing the driver experience. In this paper, we analyze the effects of mental workload and perceptual load on psychophysiological dimensions and provide a machine learning-based framework for mental and perceptual load estimation in a dual task scenario for in-vehicle interaction (https://github.com/amrgomaaelhady/MWL-PL-estimator). We use off-the-shelf non-intrusive sensors that can be easily integrated into the vehicle's system. Our statistical analysis shows that while mental workload influences some psychophysiological dimensions, perceptual load shows little effect. Furthermore, we classify the mental and perceptual load levels through the fusion of these measurements, moving towards a real-time adaptive invehicle interface that is personalized to user behavior and driving conditions. We report up to 89% mental workload classification accuracy and provide a real-time minimally-intrusive solution. CCS CONCEPTS• Human-centered computing → User centered design; • Applied computing → Psychology; • Computing methodologies → Classification and regression trees.

show abstract

Section: Hmi Design In Automotive Domainmentioning

confidence: 99%

What’s on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving

Gomaa

Alles

Meiser

et al. 2022

Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…This work proposes a dual-task scenario of outside-the-vehicle object referencing while driving. A machine learning approach versus a neural network one and early versus late fusion techniques are compared in [12], which will be expanded with more focus on the adaptation and continuous learning aspects.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Lastly, multiple possible machine learning solutions (depending on the previous factors) are utilized and planned for this work. While preliminary results are published in [12], several expansions are planned regarding learning models, fusion approaches, and timing analysis.…”

Section: Multimodal Fusionmentioning

confidence: 99%

“…More specifically, I first present initial results on investigating the specific characteristics of each modality and the interaction between them, and highlight individual differences (as published in [11]). Second, I show a learning-based multimodal fusion approach for referencing using machine learning algorithms and highlight the first step for user-specific adaptation techniques using transfer-of-learning (as published in [12]). Finally, I propose a continuous learning approach and an open-source framework for model adaptation based on the driver's implicit and explicit feedback in the object referencing task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces

Gomaa

2022

Proceedings of the 2022 International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

With the recently increasing capabilities of modern vehicles, novel approaches for interaction emerged that go beyond traditional touch-based and voice command approaches. Therefore, hand gestures, head pose, eye gaze, and speech have been extensively investigated in automotive applications for object selection and referencing. Despite these significant advances, existing approaches mostly employ a one-model-fits-all approach unsuitable for varying user behavior and individual differences. Moreover, current referencing approaches either consider these modalities separately or focus on a stationary situation, whereas the situation in a moving vehicle is highly dynamic and subject to safety-critical constraints. In this paper, I propose a research plan for a user-centered adaptive multimodal fusion approach for referencing external objects from a moving vehicle. The proposed plan aims to provide an open-source framework for user-centered adaptation and personalization using user observations and heuristics, multimodal fusion, clustering, transfer-of-learning for model adaptation, and continuous learning, moving towards trusted human-centered artificial intelligence. CCS CONCEPTS• Human-centered computing → User centered design; HCI theory, concepts and models; HCI design and evaluation methods;• Computing methodologies → Neural networks.

show abstract

“…Gomaa et al study gaze and pointing modality for the driver's behavior while pointing to outside objects [11]. They further proposed various machine learning methods, including deep neural networks for a personalized fusion to enhance the predictions [12]. Aftab et al demonstrate how multiple inputs, namely, head pose, gaze and finger pointing gesture, enhance the predictions of the driver's pointing direction for object selection inside the vehicle [1] and what limitations arise when pointing to objects outside the vehicle [2].…”

Section: Related Workmentioning

confidence: 99%

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Aftab

Beeck

2022

27th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle

Cited by 8 publications

References 36 publications

What’s on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving

What’s on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving

Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Contact Info

Product

Resources

About