A generic noninvasive neuromotor interface for human-computer interaction

Abstract

Since the advent of computing, humans have sought computer input technologies that are expressive, intuitive, and universal. While diverse modalities have been developed, including keyboards, mice, and touchscreens, they require interaction with an intermediary device that can be limiting, especially in mobile scenarios. Gesture-based systems utilize cameras or inertial sensors to avoid an intermediary device, but they tend to perform well only for unobscured or overt movements. Brain computer interfaces (BCIs) have been imagined for decades to solve the interface problem by allowing for input to computers via thought alone. However high-bandwidth communication has only been demonstrated using invasive BCIs with decoders designed for single individuals, and so cannot scale to the general public. In contrast, neuromotor signals found at the muscle offer access to subtle gestures and force information. Here we describe the development of a noninvasive neuromotor interface that allows for computer input using surface electromyography (sEMG). We developed a highly-sensitive and robust hardware platform that is easily donned/doffed to sense myoelectric activity at the wrist and transform intentional neuromotor commands into computer input. We paired this device with an infrastructure optimized to collect training data from thousands of consenting participants, which allowed us to develop generic sEMG neural network decoding models that work across many people without the need for per-person calibration. Test users not included in the training set demonstrate closed-loop median performance of gesture decoding at 0.5 target acquisitions per second in a continuous navigation task, 0.9 gesture detections per second in a discrete gesture task, and handwriting at 17.0 adjusted words per minute. We demonstrate that input bandwidth can be further improved up to 30% by personalizing sEMG decoding models to the individual, anticipating a future in which humans and machines co-adapt to provide seamless translation of human intent. To our knowledge this is the first high-bandwidth neuromotor interface that directly leverages biosignals with performant out-of-the-box generalization across people.

Big Data in Myoelectric Control: Large Multi-User Models Enable Robust Zero-Shot EMG-based Discrete Gesture Recognition

“…The adoption of highly separable wake gestures (akin to common voice prompts, such as "Hey Google") may also alleviate some of these issues (Kumar et al, 2021;Eddy et al, 2024a), although this may represent a missed opportunity for subtle, always-available ubiquitous input. Even in the recent white paper by Ctrl Labs (Labs et al, 2024), this issue of out-of-set input was not addressed, as all of the control tasks occurred for dedicated tasks. Correspondingly, exploring techniques to improve confidence distributions, such as open set (Bendale and Boult, 2016;Geng et al, 2020) and outlier detection techniques (Zhao et al, 2019), to enable the rejection of incorrect gestures, unknown gestures, and ADLs, is a crucial direction for future research.…”

“…Despite recent demonstrations suggesting that cross-user models perform reasonably well with large amounts of data (Labs et al, 2024;Vásconez et al, 2023;Valdivieso Caraguay et al, 2023;Barona López et al, 2024), many questions remain unanswered. This work thus explores cross-user myoelectric control practices for discrete event-driven systems to uncover and guide future research directions and highlight best practices moving forward.…”

“…Due to non-standardized hardware and data collection protocols, many researchers endeavour to collect study-specific datasets that either never get publicly released or cannot be pooled with other pre-recorded data. In particular, other than the 612 user dataset used in this study (Benalcazar et al, 2021), and the private collection done by Ctrl Labs (Labs et al, 2024), very few large (>100 user) EMG datasets have been recorded, and almost none are publicly available. This is contrary to other machine learning fields, whereby large datasets have become commonplace.…”

“…This came around the same time that Ctrl Labs first revealed their technology to the research community, enabling neuromuscular control (i.e., myoelectric control) (Melcer et al, 2018). Recently, after nearly five years of closed-door development, Ctrl Labs (now Meta) has claimed that zero-shot cross-user myoelectric control is possible for a variety of applications, including (1) 1D cursor control, (2) discrete control through thumb swipes and (3) continuous handwriting (Labs et al, 2024). Unlike the majority of EMG research, this work was conducted by a mega-cap multi-national company, a team of ∼200 individual contributors, a closed-source 16-channel EMG armband worn on the wrist, and data collections consisting of thousands of participants.…”

“…In particular, as opposed to the continuous control schemes used for prosthesis control, some HCI practitioners began to explore the recognition of gesture sequences (e.g., finger taps or snaps) that could be mapped to discrete inputs, as these aligned better with common inputs to a computer system like button clicks Saponas et al (2009). Instead of continuously making predictions on static contractions, these so-called discrete myoelectric control systems make a single prediction at the end of a dynamic gesture (e.g., a swipe of the wrist), meaning it is a many-to-one mapping between windows of EMG data and predictions from the classifier (Eddy et al, 2023c;Labs et al, 2024).…”