Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects

Aftab, Abdul Rafey; Beeck, Michael von der; Rohrhirsch, Steven; Diotte, Benoît; Feld, Michael S.

doi:10.1109/iv48863.2021.9575815

Cited by 2 publications

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consequently, Moniri et al (Moniri & Müller, 2012) studied the single task of outside-thevehicle referencing from the passenger seat using pointing, head pose, and eye gaze. Similarly, Aftab et al (Aftab et al, 2021) combined these modalities using a late fusion approach based on a neural network to reference objects from a stationary vehicle. While these approaches showed great promise, they still considered only a subsymbolic method for adaptation with a focus on data-driven approaches and did not consider user-specific behavior further.…”

Section: Background and Related Workmentioning

confidence: 99%

Towards Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems

Gomaa,

Feld

2023

International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Recent advances in machine learning, particularly deep learning, have enabled autonomous systems to perceive and comprehend objects and their environments in a perceptual subsymbolic manner. These systems can now perform object detection, sensor data fusion, and language understanding tasks. However, there is a growing need to enhance these systems to understand objects and their environments more conceptually and symbolically. It is essential to consider both the explicit teaching provided by humans (e.g., describing a situation or explaining how to act) and the implicit teaching obtained by observing human behavior (e.g., through the system's sensors) to achieve this level of powerful artificial intelligence. Thus, the system must be designed with multimodal input and output capabilities to support implicit and explicit interaction models. In this position paper, we argue for considering both types of inputs, as well as human-in-the-loop and incremental learning techniques, for advancing the field of artificial intelligence and enabling autonomous systems to learn like humans. We propose several hypotheses and design guidelines and highlight a use case from related work to achieve this goal.

show abstract

Section: Background and Related Workmentioning

confidence: 99%