A Hierarchical Approach for Associating Body-Worn Sensors to Video Regions in Crowded Mingling Scenarios

Abstract

We address the complex problem of associating several wearable devices with the spatio-temporal region of their wearers in video during crowded mingling events using only acceleration and proximity. This is a particularly important first step for multi-sensor behavior analysis using video and wearable technologies, where the privacy of the participants must be maintained. Most state-of-the-art works using these two modalities perform their association manually, which becomes practically unfeasible as the number of people in the scene increases. We proposed an automatic association method based on a hierarchical linear assignment optimization, which exploits the spatial context of the scene. Moreover, we present extensive experiments on matching from 2 to more than 69 acceleration and video streams, showing significant improvements over a random baseline in a real world crowded mingling scenario. We also show the effectiveness of our method for incomplete or missing streams (up to a certain limit) and analyze the trade-off between length of the streams and number of participants. Finally, we provide an analysis of failure cases, showing that deep understanding of the social actions within the context of the event is necessary to further improve performance on this intriguing task.

Person Re-ID by Fusion of Video Silhouettes and Wearable Signals for Home Monitoring Applications

“…A comparison between the three tested architectures reveals that the best performances are obtained by the fully-conv model when trained using our novel RTL, with an AVG auROC of 76.3%. The full ROC curves for this particular model are presented in Figure 6 c, together with the ROC curves for the baseline method of Shigeta et al [ 38 ] in Figure 6 a and Cabrera-Quiros et al [ 42 ] in Figure 6 b, which only manage to achieve AVG auROCs of 55.6% and 52.1%, respectively. In terms of best AVG auROC, the best model was trained on a combination of Easy and Hard negatives; while adopting harder negatives samples during training may improve the performances for SSSA auROC, the degradation over other scores leads to a lower AVG auROC.…”

“…The results of these experiments are presented in Figure 8 , which shows that our method outperforms the baselines for every case studied, in spite of a degradation of performances when the number of people increases from 2 to 10. Despite Cabrera et al [ 42 ] being explicitly designed to deal with mingling events and crowded scenes, when their algorithm is applied to the short video clips, their performances drastically drop, with results that are almost on par with random guesses in the hardest scenario (i.e., DSSA, SSSA in Figure 8 c).…”

“…All these methods are limited to cases where the wearable device is in the line of sight to the camera, which is not a reliable proposition. In [ 41 , 42 ], Cabrera-Quiros et al tackle the case of crowd-mingling events that include dozens of participants, recorded by cameras, accelerometers and proximity sensors. They estimate acceleration from the video optical flow and use the measurements from proximity sensors to cluster neighbouring people and hierarchically associate them to wearables.…”

ConfLab: A Rich Multimodal Multisensor Dataset of Free-Standing Social Interactions in the Wild

“…A crucial assumption made in many former multimodal datasets [1,6,7] is that the association of video data to the wearable modality can be manually performed. Few works [37,38] have tried to address this issue but using movement cues alone to associate the modalities is challenging as conversing individuals are mostly stationary. This remains a significant and open question for future large scale deployable multimodal systems.…”

“…In practice, it is preferable to avoid this step by using a fully automated multimodal association approach. However this remains an open research challenge [37,38]. During the event, participants mingled freely-they were allowed to carry bags or use mobile phones.…”