Singer Identification for Metaverse with Timbral and Middle-Level Perceptual Features

Abstract

Metaverse is an interactive world that combines reality and virtuality, where participants can be virtual avatars. Anyone can hold a concert in a virtual concert hall, and users can quickly identify the real singer behind the virtual idol through the singer identification. Most singer identification methods are processed using the frame-level features. However, expect the singer's timbre, the music frame includes music information, such as melodiousness, rhythm, and tonal. It means the music information is noise for using frame-level features to identify the singers. In this paper, instead of only the frame-level features, we propose to use another two features that address this problem. Middle-level feature, which represents the music's melodiousness, rhythmic stability, and tonal stability, and is able to capture the perceptual features of music. The timbre feature, which is used in speaker identification, represents the singers' voice features. Furthermore, we propose a convolutional recurrent neural network (CRNN) to combine three features for singer identification. The model firstly fuses the frame-level feature and timbre feature and then combines middle-level features to the mix features. In experiments, the proposed method achieves comparable performance on an average F1 score of 0.81 on the benchmark dataset of Artist20, which significantly improves related works.

A Review of Metaverse’s Definitions, Architecture, Applications, Challenges, Issues, Solutions, and Future Trends

“…Because of this, our digital prints and marks as well as digital bread crumbs are being followed, and they have the potential to expose the current locations we are in, age, preferred shopping, friends, preferred movies, and many more. In addition, this type of tracking could lead to aggressive entities gaining access to top secret information about oneself [123,124], including credit card details [125], social protection identification digits [126], the name of one's mother's maiden name, someone health records, and information about someone bank account [127][128][129] and among other things. This data is put at risk in a variety of ways that are all connected in some manner, and social engineering is one of them.…”