Deep Multi-modal Representation Schemes for Federated 3D Human Action Recognition

Abstract

The present work investigates the problem of multi-modal 3D human action recognition in a holistic way, following the recent and highly promising trend within the context of Deep Learning (DL), the so-called 'Federated Learning' (FL) paradigm. In particular, novel contributions of this work include: a) a methodology for enabling the incorporation of depth and 3D flow information in DL action recognition schemes, b) multiple modality fusion schemes that operate at different levels of granularity (early, slow, late), and c) federated aggregation mechanisms for adaptively guiding the action recognition learning process, by realizing cross-domain knowledge transfer in a distributed manner. A new large-scale multi-modal multi-view 3D action recognition dataset is also introduced, which involves a total of 132 human subjects. Extensive experiments provide a detailed analysis of the problem at hand and demonstrate the particular characteristics of the involved uni/multimodal representation schemes in both centralized and distributed scenarios. It is observed that the proposed FL multi-modal schemes achieve acceptable recognition performance in the proposed dataset in two challenging data distribution scenarios.

FedLID: Self-Supervised Federated Learning for Leveraging Limited Image Data

“…FL is an innovative ML approach that leverages decentralized data and computational resources to deliver more tailored and flexible applications while upholding the privacy of users and organizations. FL has demonstrated exceptional results in numerous visual analysis tasks, such as image classification, object detection and action recognition [25,12,32], indicating its robustness and effectiveness in these areas.…”

Fostering Trustworthiness of Federated Learning Ecosystem through Realistic Scenarios

“…By utilizing FL, it is possible to uphold data protection laws and regulations, thereby ensuring that privacy is not compromised. FL has demonstrated exceptional results in numerous analysis tasks, such as image classification, object detection, and action recognition [1][2][3], indicating its robustness and effectiveness in these areas. To elaborate further, FL allows data to remain on individual devices (cross-device) or servers (cross-silo) rather than being centralized, thereby avoiding potential privacy breaches.…”