New Interfaces for Classifying Performance Gestures in Music

Climent

2020

Entropy

Self Cite

Interactive music uses wearable sensors (i.e., gestural interfaces—GIs) and biometric datasets to reinvent traditional human–computer interaction and enhance music composition. In recent years, machine learning (ML) has been important for the artform. This is because ML helps process complex biometric datasets from GIs when predicting musical actions (termed performance gestures). ML allows musicians to create novel interactions with digital media. Wekinator is a popular ML software amongst artists, allowing users to train models through demonstration. It is built on the Waikato Environment for Knowledge Analysis (WEKA) framework, which is used to build supervised predictive models. Previous research has used biometric data from GIs to train specific ML models. However, previous research does not inform optimum ML model choice, within music, or compare model performance. Wekinator offers several ML models. Thus, we used Wekinator and the Myo armband GI and study three performance gestures for piano practice to solve this problem. Using these, we trained all models in Wekinator and investigated their accuracy, how gesture representation affects model accuracy and if optimisation can arise. Results show that neural networks are the strongest continuous classifiers, mapping behaviour differs amongst continuous models, optimisation can occur and gesture representation disparately affects model mapping behaviour; impacting music practice.

Section: Resultsmentioning

confidence: 99%

New Interfaces and Approaches to Machine Learning When Classifying Gestures within Music

Climent

2020

Entropy

Self Cite

“…In [9], performance technique is investigated on the violin instrument using orientation data (from Myo armbands) within set conditions, such as simple musical phrases and set windows of time, in order to observe, structure and understand the data. Our previous work [25][26][27] investigated how musical gestures for the piano (and gestures not using instruments ('in the air')) can be classified using biometric data, from the Myo armbands, with varying levels of efficacy, using continuous…”

Section: Classifying Musical Gesturementioning

confidence: 99%

“…Within the technology industry, Apple and Facebook were recently granted patents for using biometrics within wearable technologies towards individualised feedback outside of music [23,24]. In this work, we build on our previous research which classified musical gestures using EMG data from Myo armbands and compared the classification efficacy/behaviour of several models using Wekinator [25][26][27] by adopting similar data acquisition/processing methods, albeit with a different focus. In this work, we aim to solve the above motivations by developing a method which will allow us to observe if an AI guitar tutor can be created.…”

mentioning

confidence: 99%

Towards Developing a Virtual Guitar Instructor through Biometrics Informed Human-Computer Interaction

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Jay

et al. 2023

Self Cite

Fig. 1. Capturing guitar performance information via our developed multimodal system, operating across two networked computers (accommodating two Myo wearable sensors, a microphone and a video camera).Within the last few years, wearable sensor technologies have allowed us to access novel biometrics that give us the ability to connect musical gesture to computing systems. Doing this affords us to study how we perform musically and understand the process at data level. However, biometric information is complex and cannot be directly mapped to digital systems. In this work, we study how guitar performance techniques can be captured/analysed towards developing an AI which can provide real-time feedback to guitar students.We do this by performing musical exercises on the guitar whilst acquiring and processing biometric (plus audiovisual) information during their performance. Our results show: there are notable differences within biometrics when playing a guitar scale in two different ways (legato and staccato) and this outcome can be used to motivate our intention to build an AI guitar tutor.CCS Concepts: • Human-centered computing → Gestural input.

“…This is because the gestural signal onset can be either omitted or included in the ML model training process and affect musical intention. Previous research has looked at the efficacy of ML models in Wekinator on two musical gestures used in piano practice [22] and has extended this investigation with further gestures, including model optimisation, plus the effect of including gestural signal onset on model accuracy/efficacy [23]. Our study is informed by this prior research.…”

Section: Introductionmentioning

confidence: 99%

Classifying Biometric Data for Musical Interaction Within Virtual Reality

Lecture Notes in Computer Science

Climent

2022

Self Cite

Since 2015, commercial gestural interfaces have widened accessibility for researchers and artists to use novel Electromyographic (EMG) biometric data. EMG data measures musclar amplitude and allows us to enhance Human-Computer Interaction (HCI) through providing natural gestural interaction with digital media. Virtual Reality (VR) is an immersive technology capable of simulating the real world and abstractions of it. However, current commercial VR technology is not equipped to process and use biometric information. Using biometrics within VR allows for better gestural detailing and use of complex custom gestures, such as those found within instrumental music performance, compared to using optical sensors for gesture recognition in current commercial VR equipment. However, EMG data is complex and machine learning must be used to employ it. This study uses a Myo armband to classify four custom gestures in Wekinator and observe their prediction accuracies and representations (including or omitting signal onset) to compose music within VR. Results show that specific regression and classification models, according to gesture representation type, are the most accurate when classifying four music gestures for advanced music HCI in VR. We apply and record our results, showing that EMG biometrics are promising for future interactive music composition systems in VR.