A Novel Method of Combining Computer Vision, Eye-Tracking, EMG, and IMU to Control Dexterous Prosthetic Hand

Shi, Chunyuan; Le, Qi; Yang, Dapeng; Zhao, Jingdong; Liu, Hong

doi:10.1109/robio49542.2019.8961582

Cited by 10 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IMU information has also been used for biomechanical analysis [56]- [60], such as gait analysis, which can be used to evaluate motor recovery. It can also be used with assistive devices [61]- [64] and in combination with other sensors, such as EMG sensors [5], [7]. Features extracted from EMG signals have been used as inputs of the control loop of rehabilitation robots [65]- [68].…”

Section: Discussionmentioning

confidence: 99%

A Versatile Embedded Platform for Implementation of Biocooperative Control in Upper-Limb Neuromotor Rehabilitation Scenarios

et al. 2023

View full text Add to dashboard Cite

Biocooperative control uses both biomechanical and physiological information of the user to achieve a reliable human-robot interaction. In the context of neuromotor rehabilitation, such control can enhance rehabilitation experience and outcomes. However, the high cost and large volume of the commercial systems for physiological signal acquisition are major limitations for the development of such control. We present a highly versatile, low-cost and wearable embedded system that integrates the most commonly used sensors in this field: inertial measurement unit (IMU), electrocardiography (ECG), electromyography (EMG), galvanic skin response (GSR) and skin temperature (SKT) sensors. Additionally, the compact system combines wireless communication for data transmission and a high-efficiency microcontroller for real-time signal processing and control. We tested the system in two common neuromotor rehabilitation scenarios. The first is an upper-limb rehabilitation VR-based exergame, in which the patient must collect as many coins as possible. Movement recognition of the hand and arm is performed based on EMG and IMU information, respectively. The second is adaptive assistive control that adjusts the level of assistance of a wrist rehabilitation robot according to the physiological state and motor performance of the patient using GSR, ECG and SKT data. The quality of the recorded signals and the processing capacity of the system meet the needs of the two upper-limb rehabilitation applications. The wearable system is highly versatile, open, configurable and low cost, and it could promote the development of real-time biocooperative control for a wide range of neuromotor rehabilitation applications.INDEX TERMS Biocooperative control, embedded system, neuromotor rehabilitation, real-time signal processing, wearable sensors.

show abstract

Section: Discussionmentioning

confidence: 99%

A Versatile Embedded Platform for Implementation of Biocooperative Control in Upper-Limb Neuromotor Rehabilitation Scenarios

et al. 2023

View full text Add to dashboard Cite

show abstract

“…An advantage of these sensors is that they can provide information about the kinematics of human motion depending on their location on the body, and when used in combination with EMG data, they can improve the classification of gesture-based control algorithms [ 3 , 11 ]. Other advantages of using EMG–IMU sensor fusion include the ability to quantify motor function of stroke patients [ 12 ], to improve the control of prosthetic devices [ 13 , 14 ], and to account for postural changes when performing hand gestures [ 15 , 16 ]. Although promising, the use of EMG and IMU-based sensor fusion has not been fully explored in the area of user-independent classification algorithms.…”

Section: Introductionmentioning

confidence: 99%

User-Independent Hand Gesture Recognition Classification Models Using Sensor Fusion

Alfaro

Trejos

2022

Sensors

View full text Add to dashboard Cite

Recently, it has been proven that targeting motor impairments as early as possible while using wearable mechatronic devices for assisted therapy can improve rehabilitation outcomes. However, despite the advanced progress on control methods for wearable mechatronic devices, the need for a more natural interface that allows for better control remains. To address this issue, electromyography (EMG)-based gesture recognition systems have been studied as a potential solution for human–machine interface applications. Recent studies have focused on developing user-independent gesture recognition interfaces to reduce calibration times for new users. Unfortunately, given the stochastic nature of EMG signals, the performance of these interfaces is negatively impacted. To address this issue, this work presents a user-independent gesture classification method based on a sensor fusion technique that combines EMG data and inertial measurement unit (IMU) data. The Myo Armband was used to measure muscle activity and motion data from healthy subjects. Participants were asked to perform seven types of gestures in four different arm positions while using the Myo on their dominant limb. Data obtained from 22 participants were used to classify the gestures using three different classification methods. Overall, average classification accuracies in the range of 67.5–84.6% were obtained, with the Adaptive Least-Squares Support Vector Machine model obtaining accuracies as high as 92.9%. These results suggest that by using the proposed sensor fusion approach, it is possible to achieve a more natural interface that allows better control of wearable mechatronic devices during robot assisted therapies.

show abstract

“…A multimodal system was proposed by Shi et al ( 2019 ) combining eye tracking, CV, sEMG, and an Inertial Measurement Unit (IMU) integrated into the HIT AID Hand prosthetic device. The Kinect 2.0 (Microsoft, USA), a 3D Camera, collected color, depth, and infrared scene images from the user's perspective.…”

Section: Introductionmentioning

confidence: 99%

A Hybrid 3D Printed Hand Prosthesis Prototype Based on sEMG and a Fully Embedded Computer Vision System

2022

View full text Add to dashboard Cite

This study presents a new approach for an sEMG hand prosthesis based on a 3D printed model with a fully embedded computer vision (CV) system in a hybrid version. A modified 5-layer Smaller Visual Geometry Group (VGG) convolutional neural network (CNN), running on a Raspberry Pi 3 microcomputer connected to a webcam, recognizes the shape of daily use objects, and defines the pattern of the prosthetic grasp/gesture among five classes: Palmar Neutral, Palmar Pronated, Tripod Pinch, Key Grasp, and Index Finger Extension. Using the Myoware board and a finite state machine, the user's intention, depicted by a myoelectric signal, starts the process, photographing the object, proceeding to the grasp/gesture classification, and commands the prosthetic motors to execute the movements. Keras software was used as an application programming interface and TensorFlow as numerical computing software. The proposed system obtained 99% accuracy, 97% sensitivity, and 99% specificity, showing that the CV system is a promising technology to assist the definition of the grasp pattern in prosthetic devices.

show abstract

A Novel Method of Combining Computer Vision, Eye-Tracking, EMG, and IMU to Control Dexterous Prosthetic Hand

Cited by 10 publications

References 9 publications

A Versatile Embedded Platform for Implementation of Biocooperative Control in Upper-Limb Neuromotor Rehabilitation Scenarios

A Versatile Embedded Platform for Implementation of Biocooperative Control in Upper-Limb Neuromotor Rehabilitation Scenarios

User-Independent Hand Gesture Recognition Classification Models Using Sensor Fusion

A Hybrid 3D Printed Hand Prosthesis Prototype Based on sEMG and a Fully Embedded Computer Vision System

Contact Info

Product

Resources

About