Evaluation Strategy of the Piano Performance by the Deep Learning Long Short-Term Memory Network

Chang, Xunyun; Peng, Liangqing

doi:10.1155/2022/6727429

Cited by 8 publications

(8 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most widely used method for piano skill evaluation is based on the aural mode. Chang et al [1] proposed an LSTM-based piano performance evaluation strategy for evaluating piano performance. Their approach incorporates three indicators, namely overall evaluation, rhythm, and expressiveness.…”

Section: Piano Performance In Unimodalitymentioning

confidence: 99%

“…X m [:, X v .shape [1] :] ← X a ; return X m ; 12: end function Aggregation option: During the piano performance, the score obtained by the players can be perceived as an additive operation. It is often advantageous to perform linear operations on the learned features, which enhances the interpretability and expressiveness of the learned features.…”

Section: Algorithm 1 Model Initialization Algorithmmentioning

confidence: 99%

“…At present, most studies on piano performance evaluation have been conducted based on a single audio mode [1][2][3], disregarding the information contained in the video mode, such as playing technique and playing posture, resulting in a one-sided assessment of the player's skill level and an inability to conduct a comprehensive assessment from multiple aspects.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation

2023

View full text Add to dashboard Cite

With the rise in piano teaching in recent years, many people have joined the ranks of piano learners. However, the high cost of traditional manual instruction and the exclusive one-on-one teaching model have made learning the piano an extravagant endeavor. Most existing approaches, based on the audio modality, aim to evaluate piano players’ skills. Unfortunately, these methods overlook the information contained in videos, resulting in a one-sided and simplistic evaluation of the piano player’s skills. More recently, multimodal-based methods have been proposed to assess the skill level of piano players by using both video and audio information. However, existing multimodal approaches use shallow networks to extract video and audio features, which limits their ability to extract complex spatio-temporal and time-frequency characteristics from piano performances. Furthermore, the fingering and pitch-rhythm information of the piano performance is embedded within the spatio-temporal and time-frequency features, respectively. Therefore, we propose a ResNet-based audio-visual fusion model that is able to extract both the visual features of the player’s finger movement track and the auditory features, including pitch and rhythm. The joint features are then obtained through the feature fusion technique by capturing the correlation and complementary information between video and audio, enabling a comprehensive and accurate evaluation of the player’s skill level. Moreover, the proposed model can extract complex temporal and frequency features from piano performances. Firstly, ResNet18-3D is used as the backbone network for our visual branch, allowing us to extract feature information from the video data. Then, we utilize ResNet18-2D as the backbone network for the aural branch to extract feature information from the audio data. The extracted video features are then fused with the audio features, generating multimodal features for the final piano skill evaluation. The experimental results on the PISA dataset show that our proposed audio-visual fusion model, with a validation accuracy of 70.80% and an average training time of 74.02 s, outperforms the baseline model in terms of performance and operational efficiency. Furthermore, we explore the impact of different layers of ResNet on the model’s performance. In general, the model achieves optimal performance when the ratio of video features to audio features is balanced. However, the best performance achieved is 68.70% when the ratio differs significantly.

show abstract

Section: Piano Performance In Unimodalitymentioning

confidence: 99%

Section: Algorithm 1 Model Initialization Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation

2023

View full text Add to dashboard Cite

show abstract

“…is example uses a weighted average algorithm to calculate the updated model parameters after collecting training parameters from multiple nodes. en, the updated parameters are made into broadcast parameters and passed to each node [22]. e principle of decentralization is that multiple nodes are connected in pairs and can update parameters with each other.…”

Section: Backpropagation Algorithmmentioning

confidence: 99%

Educational Evaluation of Piano Performance by the Deep Learning Neural Network Model

Liao

2022

Mobile Information Systems

View full text Add to dashboard Cite

In recent years, the piano education industry has occupied a huge market. However, the automatic evaluation function of piano performance has shortcomings in existing piano education products. Deep learning (DL) algorithm and the recurrent neural network (RNN) structure can help in automatics evaluation function of the piano performance. This paper proposes a Musical Instrument Digital Interface (MIDI) piano evaluation scheme based on the RNN structure and the Spark computing engine using the Deeplearning4J DL framework. The Deeplearning4J framework can run on the Java Virtual Machine; therefore, the entire system does not require cross-platform development. The Spark distributed computing engine realizes parallelization in music data preprocessing, feature extraction, and model training. Combined with the training user interface (UI) provided by the Deeplearning4J, it can improve developmental efficiency. Additionally, the RNN parameters are analyzed. The results demonstrate that the error value of the three-layer RNN structure is smaller than other closest rivals’ techniques. In particular, few piano training institutions and MIDI website datasets are used as the basis, and the experimental samples are collected. The neural network is trained, and the performance of the evaluation model is tested. The results show that the evaluation outcomes of the designed performance evaluation model for the piano are fundamentally consistent with the real levels of the players with assured feasibility; after 3k times of the training periods, the error of the RNN model is close to 0.01 and the network converges.

show abstract

“…There is an inherently complex connection between musical properties and the emotions they express, and standard MER techniques have always had problems accurately expressing this. Based on the musician, even an identical recording of piano music may cause entirely distinct emotions, which those methods are unable to express [2] .…”

Section: Introductionmentioning

confidence: 99%

Application and algorithm optimization of music emotion recognition in piano performance evaluation

Zhang,

Cai,

Zhang

2024

Environ. Soc. Psychol.

View full text Add to dashboard Cite

In the current research, we integrate distinct learning modalities—Curriculum Learning (CL) and Reinforcement Learning (RL)—in an attempt to develop and optimize Music Emotion Recognition (MER) in piano performance. Classical approaches have never been successful when applied in the field of determining the degree of emotion in the music of the piano, owing to the substantial complexity required. Addressing this particular issue is the primary motivation for the present endeavour. In an approach that’s comparable to how human beings acquire information, it trains the RL agent CL in phases; such an approach improves the student’s learning model in understanding the diverse emotions expressed by musical compositions. A higher rating of performance can be achieved after learning the model to recognize emotions more effectively and precisely. A set of piano melodies with emotional content notes has been included in the EMOPIA repository for use when conducting the process of evaluation. In order to benchmark the proposed approach with different models, parameters including R2, Mean Absolute Error (MAE), and Root Mean Square Error (RMSE) were deployed. Studies indicate that the recommended approach accurately recognizes the emotions expressed by piano-playing music. In challenging tasks like MER, the significance of implementing the CL paradigm with the RL has been emphasized using the outcomes mentioned earlier.

show abstract

Evaluation Strategy of the Piano Performance by the Deep Learning Long Short-Term Memory Network

Cited by 8 publications

References 28 publications

A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation

A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation

Educational Evaluation of Piano Performance by the Deep Learning Neural Network Model

Application and algorithm optimization of music emotion recognition in piano performance evaluation

Contact Info

Product

Resources

About