Mirror-based ultrasound system for hand gesture classification through convolutional neural network and vision transformer

Abstract

This research presents an innovative mirror-based ultrasound system designed for hand gesture classification using Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures. Hand gesture recognition using ultrasound has garnered significant interest due to its potential applications in various fields such as prosthetic control and human-machine interfacing. Traditionally, ultrasound probes are placed perpendicular to the forearm causing discomfort and interference with natural arm movements due to the center of mass of the wearable ultrasound system being distanced from the body. To address this challenge, a novel approach utilizing the advantages of acoustic reflection is proposed. A convex ultrasound probe is strategically aligned with the forearm, and ultrasound waves are transmitted to the forearm, and received back using a mirror placed at 45 degrees to the imaging region and the forearm. By aligning the probe parallel to the arm, the center of mass is brought closer to the body, ensuring enhanced stability and reduced strain on the user's arm during data collection. A dataset comprising 5 hand gestures was collected to train and evaluate the performance with Support Vector Machines with linear kernel, CNN, and ViT based approaches. It was observed that the performance of the mirror-based ultrasound system is comparable to the traditional perpendicular approach for hand gesture classification.The experimental results demonstrate the potential of the system in assisting with data acquisition and device development for hand gesture recognition using ultrasound in the field of human-machine interfacing, prosthetic control, humancomputer interaction, and beyond.