A Multimodal Intention Detection Sensor Suite for Shared Autonomy of Upper-Limb Robotic Prostheses

Abstract

Neurorobotic augmentation (e.g., robotic assist) is now in regular use to support individuals suffering from impaired motor functions. A major unresolved challenge, however, is the excessive cognitive load necessary for the human–machine interface (HMI). Grasp control remains one of the most challenging HMI tasks, demanding simultaneous, agile, and precise control of multiple degrees-of-freedom (DoFs) while following a specific timing pattern in the joint and human–robot task spaces. Most commercially available systems use either an indirect mode-switching configuration or a limited sequential control strategy, limiting activation to one DoF at a time. To address this challenge, we introduce a shared autonomy framework centred around a low-cost multi-modal sensor suite fusing: (a) mechanomyography (MMG) to estimate the intended muscle activation, (b) camera-based visual information for integrated autonomous object recognition, and (c) inertial measurement to enhance intention prediction based on the grasping trajectory. The complete system predicts user intent for grasp based on measured dynamical features during natural motions. A total of 84 motion features were extracted from the sensor suite, and tests were conducted on 10 able-bodied and 1 amputee participants for grasping common household objects with a robotic hand. Real-time grasp classification accuracy using visual and motion features obtained 100%, 82.5%, and 88.9% across all participants for detecting and executing grasping actions for a bottle, lid, and box, respectively. The proposed multimodal sensor suite is a novel approach for predicting different grasp strategies and automating task performance using a commercial upper-limb prosthetic device. The system also shows potential to improve the usability of modern neurorobotic systems due to the intuitive control design.

Wearable MMG-Plus-One Armband: Evaluation of Normal Force on Mechanomyography (MMG) to Enhance Human-Machine Interfacing

“…Results showed a unique user-specific pattern of performance modulation of the man-machine interface and a high potential for enhancing the accuracy of the system. Future perspectives will focus on (a) collection of a clinical dataset for testing the proposed armband on a larger group of amputees to run a clinical statistical study, (b) analysis of the effect of normal force distribution on acceleration-based MMG for comparison, (c) evaluation of hand gesture detection during the transient phase of the signal, and (d) incorporation of multimodal sensing to address sensitivity to motion induced artifacts (demonstrated in [18], [19]). Finally, the system has been patented [47], providing a basis for subsequent translational work for widespread patient use.…”

“…Examples of the use of MMG technology are the evaluation of muscle fatigue [11], muscle strength [12], balance [13], muscle functions [14], or the analysis of the mechanical response of the muscle after exercise [15]. Other applications of MMG technology include the examination of neuromuscular disorders [16], and prosthetic limb/robotic control [17]- [19]. In comparison to EMG, this mechanical method (i.e., MMG) may be classically considered as a modality with a lower information transfer rate, in particular regarding the neurophysiology of the muscle activation and peripheral nervous system.…”

Biosignal-Based Human–Machine Interfaces for Assistance and Rehabilitation: A Survey

“…Another low-cost multimodal sensor was presented by Gardner et al [221]. It consists of an acoustic MMG sensor, a nine-DOF IMU sensor, and a video camera.…”

Review of Advanced Medical Telerobots

“…Another relevant application is myoelectric control of prostheses [335,336], which allows users to recover lost functionality by controlling a prosthetic robotic device with their remaining muscle activity. In [337], computer vision (for autonomous object recognition) and mechanomyography (to estimate the intended muscle activation) data are fused to conduct a shared control that predicts user intent for grasp and then realizes it. In [338], once the user establishes a pre-contact between the robotic hand and an object (manual task), the shared controller optimizes the actuation of the fingers of a robotic hand to maximize the contact between the hand and object to obtain full-contact (robotic task).…”