Optical Myography: Detecting Finger Movements by Looking at the Forearm

Abstract

One of the crucial problems found in the scientific community of assistive/rehabilitation robotics nowadays is that of automatically detecting what a disabled subject (for instance, a hand amputee) wants to do, exactly when she wants to do it, and strictly for the time she wants to do it. This problem, commonly called “intent detection,” has traditionally been tackled using surface electromyography, a technique which suffers from a number of drawbacks, including the changes in the signal induced by sweat and muscle fatigue. With the advent of realistic, physically plausible augmented- and virtual-reality environments for rehabilitation, this approach does not suffice anymore. In this paper, we explore a novel method to solve the problem, which we call Optical Myography (OMG). The idea is to visually inspect the human forearm (or stump) to reconstruct what fingers are moving and to what extent. In a psychophysical experiment involving ten intact subjects, we used visual fiducial markers (AprilTags) and a standard web camera to visualize the deformations of the surface of the forearm, which then were mapped to the intended finger motions. As ground truth, a visual stimulus was used, avoiding the need for finger sensors (force/position sensors, datagloves, etc.). Two machine-learning approaches, a linear and a non-linear one, were comparatively tested in settings of increasing realism. The results indicate an average error in the range of 0.05–0.22 (root mean square error normalized over the signal range), in line with similar results obtained with more mature techniques such as electromyography. If further successfully tested in the large, this approach could lead to vision-based intent detection of amputees, with the main application of letting such disabled persons dexterously and reliably interact in an augmented-/virtual-reality setup.

VITA—an everyday virtual reality setup for prosthetics and upper-limb rehabilitation

“…Lastly, let us not forget that the intent-detection system we use is modular and can be used with several different input devices other than sEMG-based (e.g., optical- [33] or tactile-based [34]) to improve the quality and dexterity of the detection. All of these devices are not limited to controlling a hand in a VR environment.…”

Control Methods for Transradial Prostheses Based on Remnant Muscle Activity and Its Relationship with Proprioceptive Feedback

“…Nissler et al studied the potential use of an optical myography (OMG) approach for hand gesture recognition, in which a single low-resolution camera (1280 × 720 pixels) monitors the forearm skin surface deformation caused by the underlying muscle contraction using fiducial marker-based tracking methods [ 125 , 126 ]. The human skin offers very little texture, and therefore it is challenging to register relatively small deformations using visual information.…”

“…( b )—example of data acquisition during the experiment, visual fiducial markers are placed on the ventral side of the user’s forearm. Adapted from [ 126 ], licensed under CC BY 4.0.…”

Force Myography Signal-Based Hand Gesture Classification for the Implementation of Real-Time Control System to a Prosthetic Hand

“…Recently, FMG based interfacing system is also gaining attention for the utilization of prosthetic devices, the technique has been studied and tested by multiple research groups [4,5]. A comparative study between FMG and EMG has also been conducted [6][7][8]. One of the advantages of FMG over EMG is that FMG signals have shown a higher overall stability overtime.…”