Feeling the beat: a smart hand exoskeleton for learning to play musical instruments

Abstract

Individuals who have suffered neurotrauma like a stroke or brachial plexus injury often experience reduced limb functionality. Soft robotic exoskeletons have been successful in assisting rehabilitative treatment and improving activities of daily life but restoring dexterity for tasks such as playing musical instruments has proven challenging. This research presents a soft robotic hand exoskeleton coupled with machine learning algorithms to aid in relearning how to play the piano by ‘feeling’ the difference between correct and incorrect versions of the same song. The exoskeleton features piezoresistive sensor arrays with 16 taxels integrated into each fingertip. The hand exoskeleton was created as a single unit, with polyvinyl acid (PVA) used as a stent and later dissolved to construct the internal pressure chambers for the five individually actuated digits. Ten variations of a song were produced, one that was correct and nine containing rhythmic errors. To classify these song variations, Random Forest (RF), K-Nearest Neighbor (KNN), and Artificial Neural Network (ANN) algorithms were trained with data from the 80 taxels combined from the tactile sensors in the fingertips. Feeling the differences between correct and incorrect versions of the song was done with the exoskeleton independently and while the exoskeleton was worn by a person. Results demonstrated that the ANN algorithm had the highest classification accuracy of 97.13% ± 2.00% with the human subject and 94.60% ± 1.26% without. These findings highlight the potential of the smart exoskeleton to aid disabled individuals in relearning dexterous tasks like playing musical instruments.

A New Method to Evaluate Pressure Distribution Using a 3D-Printed C2-C3 Cervical Spine Model with an Embedded Sensor Array

“…Regarding the sensor's performance, there is potential for improvement in its spatial resolution that would further enhance the design of cages [46,47]. This would most likely fix the steep pressure gradients that were present.…”

Biohybrid Robotic Hand to Investigate Tactile Encoding and Sensorimotor Integration

“…To remedy these kinds of debilitating conditions, invasive BMIs have been used to connect the brain with assistive devices to enable sensation and motor control. However, restoring motor control and sensation from an assistive device in a nearly natural manner remains a scientific holy grail due to the complexity of the problem [2,3]. Nevertheless, noteworthy progress has been made toward this goal.…”

High-Performance Four-Channel Tactile Sensor for Measuring the Magnitude and Orientation of Forces

“…In recent years, the application of sensors in smart robotic hands has gained widespread attention. Reference [36] describes a technology using 3D-printed PVA stents and hydrogel casting to manufacture an exoskeleton hand capable of accurately distinguishing music performance. Another study [37] equips a robotic hand with tactile sensors to provide tactile feedback, offering amputees the possibility of possessing highly dexterous prosthetic limbs.…”