Integrated Sensor Fusion and Multi-Modal Hierarchical Neural Network for Activity Recognition in Lower Limb Prosthetics

Abstract

Activity recognition plays pivotal role in enhancing functionality for prosthetic devices, ensuring seamless integration with users' movements. However, the complexity arises from the diverse data sources, including acceleration, angular velocity, joint angles, orientation, electromyography (EMG), and marker data, necessitating a robust approach to overcome challenges in information integration. The primary challenge lies in the effective utilization of multiple sensor modalities, each with unique characteristics and potential noise sources. The proposed solution addresses this by employing advanced sensor fusion techniques, such as Kalman filtering, during data collection. Synchronization and resampling ensure temporal consistency, while noise reduction techniques, such as low-pass filters, mitigate signal distortions. To further refine the process, a hybrid optimization-based feature selection Adaptive Step Size in Marine Predators Algorithm (ASSMPA) is introduced, focusing on marker data features. ASSMPA synergizes Marine Predators Algorithm (MPA) and Pathfinder Algorithm (PFA) for optimal feature selection in marine predator pathfinding tasks. The Feature Fusion step integrates attention mechanisms to dynamically weigh the significance of different sensor modalities during the fusion process. This strategic fusion enhances the overall performance of the Multi-Modal Hierarchical Neural Network (MMHNN). The proposed model is implemented using Python.