MetaMGC: a music generation framework for concerts in metaverse

Abstract

In recent years, there has been a national craze for metaverse concerts. However, existing meta-universe concert efforts often focus on immersive visual experiences and lack consideration of the musical and aural experience. But for concerts, it is the beautiful music and the immersive listening experience that deserve the most attention. Therefore, enhancing intelligent and immersive musical experiences is essential for the further development of the metaverse. With this in mind, we propose a metaverse concert generation framework — from intelligent music generation to stereo conversion and sound field design for virtual concert stages. First, combining the ideas of reinforcement learning and value functions, the Transformer-XL music generation network is improved and used in training all the music in the POP909 dataset. Experiments show that both improved algorithms have advantages over the original method in terms of objective evaluation and subjective evaluation metrics. In addition, this paper validates a neural rendering method that can be used to generate spatial audio based on a binaural-integrated neural network with a fully convolutional technique. And the purely data-driven end-to-end model performs to be more reliable compared with traditional spatial audio generation methods such as HRTF. Finally, we propose a metadata-based audio rendering algorithm to simulate real-world acoustic environments.

Intelligent Music Applications: Innovative Solutions for Musicians and Listeners

“…Although current endeavours in the metaverse emphasise visual experiences, the musical and auditory elements of such concerts hold equal significance. Jin et al (2022) introduces the MetaMGC (Music Generation Framework for Concerts in the Metaverse) model, which provides an effective and intelligent approach to implementing music generation and immersive sound field twinning for concerts in the metaverse. This comprises of three primary components: music generation, digital audio twin, and audio rendering.…”

T109 Evaluation of the GEM® Premier™ 5000 with intelligent quality management 2 (IQM®2) at Beijing Tsinghua Changgung Hospital, Tsinghua University (Beijing, China)

“…With the success of diffusion models [123] and applications like Midjourney, demonstrating high-quality, real-life image generation capabilities, creators within the Metaverse can now generate diverse types, contents, and styles of artworks through simple textual prompts. Similarly, generative works in film [43], audio [72], or poetry [93] have produced impressive content, ultimately providing users with experiences derived from reality but elevated beyond it. This revolution extends further to architectural designs [68], landscape design [7], and urban planning [186], offering exciting possibilities to create a metaverse city.…”

Evaluating Influencing Factors of Audiences’ Attitudes toward Virtual Concerts: Evidence from China

“…Moreover, the integration of virtual stages, avatars, video games, virtual social networks, and other virtual technologies such as virtual environments, Music Internet of Things (IoMusT), and singer identification and field adaptation (MetaSID) have proven to enhance the audience’s experience [ 6 , 16 , 17 ]. Previous studies have discussed several factors that contribute to audiences’ experience of virtual concerts, including the sense of connection, interactivity, immersion, and a feeling of “being there”, all of which can affect audience satisfaction and attendance intentions [ 12 , 18 , 19 , 20 ]. Hwang & Koo highlighted the positive potential change that virtual concerts could have on the music industry [ 21 ].…”

“…The definition of virtual concerts varies in the academic literature, such as livestream concerts [3][4][5], holographic concerts [6][7][8], and concerts requiring VR hardware [9,10]. In this study, a virtual concert is defined as a musical performance in which participants are projected into a digital virtual environment in the forms of virtual avatars [11,12]. These virtual concerts are mostly game-based, e.g., those found in Fortnite, Roblox, Second Life, or Minecraft, or blockchain-based, such as those in Decentraland or The Sandbox.…”