Bimodal Control of a Vision-Based Myoelectric Hand

Abstract

In this paper, we propose a novel control scheme for a vision-based prosthetic hand. To realize complex and flexible human-like hand movements, the proposed method fuses bimodal information. Combining information from surface EMG signals with object information from a vision sensor, the system can select an appropriate hand motion. The training/recognition using both sEMG signals and object images can be performed with a single deep neural network in an end-to-end manner. The bimodal sensor information enables the system to recognize the operator's intended motion with higher accuracy than that of the conventional method using only sEMG signals. In addition, the generalization ability of the network is improved, so motion recognition robustness is enhanced against abnormal data that include partly noisy or missing samples. To verify the validity of the proposed approach, we prepared a dataset that contains the sEMG signals and the object images for 10 types of grasping motions. Three kinds of experiments were conducted: comparison of the proposed method with the conventional method, examination of the recognition robustness against partly noisy or missing samples, and challenges to recognize hand motions based on raw sEMG signals. The results revealed that the proposed bimodal network achieved considerably high recognition performance.

Efficient EMG Pattern Classification-based Control Interface for a Low-Cost Sensor Fusion Controlled Exoskeleton Glove

“…Research teams at various institutions are working on control strategies for assistive devices utilizing several different sensing technologies. Many of these strategies involve electromyography (EMG) [8][9][10][11][12][13], computer vision (CV) [8,10,13], and/or voice control [11,12].…”

“…This work is very promising for the future of post-stroke gesture recognition. In [10], the authors fused sEMG and CV information into a single bimodal model in order to control a prosthetic hand. The goal of this model was to emulate the way that the human brain processes sensory information and converts it into actuated motion.…”

“…Some of the devices in previous research require expensive equipment such as eye tracking sensors [8] and high-density EMG arrays [9] which can cost hundreds or thousands of dollars USD. Others are implemented on laptop or desktop computers, limiting the range of these devices [8][9][10][11]13]. Unfortunately, this causes these devices to still be inaccessible to low-income patients or unable to assist with some tasks of daily living.…”