Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos

Khan, Shehroz S.; Shen, Ziting; Sun, Haoying; Patel, Ax; Abedi, Ali

doi:10.48550/arxiv.2109.04021

Cited by 1 publication

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main objective of our experiments is to determine the impact of fusion strategies for the probability estimates of multimodal predictors in the context of driver observation, where averaging has presumably been the most common choice for fusion at decision-level [2], [1], [9], [4]. Tables II, III and IV display balanced and unbalanced top-1 and top-5 accuracies for different fusion schemes and rare, common and all driver behaviour categories respectively.…”

Section: B Resultsmentioning

confidence: 99%

“…Note that the division by N does not change the ranking of the summed predictions, but serves to regularize the output to sum up to 1. This fusion strategies has presumably been the most popular choice for fusion at decision-level in driver observation [2], [1], [9], [4].…”

Section: Sum-fusion and Score Averagingmentioning

confidence: 99%

“…Multimodality increasingly gains attention in driver observation systems [1], [2], [3], [4]: prediction outcomes of multiple sensors complement each other due to modalityspecific strengths and weaknesses as well as different visibility (examples in Figures 1 and 2). Rising levels of automation increase human freedom, leading to drivers being engaged in distractive behaviours more often while the type of activities become increasingly diverse.…”

Section: Introduction and Related Workmentioning

confidence: 99%

“…The state-of-the-art of multimodal driver activity recognition constantly changes depending on different architecture choices, losses and classifier components [7], [8], [9], [2], [1], [10], but a large portion of such methods employ late fusion via score averaging to link the information [2], [1], [9], [4]. Multimodal fusion algorithms can be grouped depending on the point of fusion, (e.g., early-, mid-, or late-fusion) and based on the methodology (learning-and decision-based approaches).…”

Section: Introduction and Related Workmentioning

confidence: 99%

“…Among decision-level fusion techniques, averaging of the obtained Softmax scores is presumably the most common choice in driver activity recognition [2], [1], [9], [4]. In the broader fields of general machine learning and computer vision, this approach is also highly popular [11], [12], [13], [14], [15], [16], [17], [18], but other strategies, which were rather overlooked in the field of driver observation, such as the max rule [19], [14], [17], [20] or the product rule [12], [21], [19], [22], [23], [24], [25], [17], [20] also gained attention.…”

Section: Introduction and Related Workmentioning

confidence: 99%

See 4 more Smart Citations

A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding

Roitberg¹,

Peng²,

Marinov³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual recognition inside the vehicle cabin leads to safer driving and more intuitive human-vehicle interaction but such systems face substantial obstacles as they need to capture different granularities of driver behaviour while dealing with highly limited body visibility and changing illumination. Multimodal recognition mitigates a number of such issues: prediction outcomes of different sensors complement each other due to different modality-specific strengths and weaknesses. While several late fusion methods have been considered in previously published frameworks, they constantly feature different architecture backbones and building blocks making it very hard to isolate the role of the chosen late fusion strategy itself.This paper presents an empirical evaluation of different paradigms for decision-level late fusion in video-based driver observation. We compare seven different mechanisms for joining the results of single-modal classifiers which have been both popular, (e.g. score averaging) and not yet considered (e.g. ranklevel fusion) in the context of driver observation evaluating them based on different criteria and benchmark settings. This is the first systematic study of strategies for fusing outcomes of multimodal predictors inside the vehicles, conducted with the goal to provide guidance for fusion scheme selection.

show abstract