Remixing music using source separation algorithms to improve the musical experience of cochlear implant users

Abstract

Music perception remains rather poor for many Cochlear Implant (CI) users due to the users' deficient pitch perception. However, comprehensible vocals and simple music structures are well perceived by many CI users. In previous studies researchers re-mixed songs to make music more enjoyable for them, favoring the preferred music elements (vocals or beat) attenuating the others. However, mixing music requires the individually recorded tracks (multitracks) which are usually not accessible. To overcome this limitation, Source Separation (SS) techniques are proposed to estimate the multitracks. These estimated multitracks are further re-mixed to create more pleasant music for CI users. However, SS may introduce undesirable audible distortions and artifacts. Experiments conducted with CI users (N = 9) and normal hearing listeners (N = 9) show that CI users can have different mixing preferences than normal hearing listeners. Moreover, it is shown that CI users' mixing preferences are user dependent. It is also shown that SS methods can be successfully used to create preferred re-mixes although distortions and artifacts are present. Finally, CI users' preferences are used to propose a benchmark that defines the maximum acceptable levels of SS distortion and artifacts for two different mixes proposed by CI users.

Design and Evaluation of a Real-Time Audio Source Separation Algorithm to Remix Music for Cochlear Implant Users

“…The data set was divided into the training and the testing data set with 100 and 50 songs, respectively. • Buyens Shared Data Set: As the third data set, six popular music pieces (Buyens et al, 2014) that have been used in previous CI studies to create and report a benchmark (Pons et al, 2016;Gajȩcki and Nogueira, 2018) have been also used in this study. • Custom Data Set with Virtual Acoustics: All the previous data sets were studio recordings with no spatial characteristics.…”

“…Previous research in the area of music enhancement for CI users has focused on reducing music complexity (Nagathil et al, 2017) or on amplifying vocals relative to the background instruments (Buyens et al, 2014;Pons et al, 2016;Gajȩcki and Nogueira, 2018). Spectral complexity reduction of music was investigated based on dimensionality reduction techniques, such as principal component analysis and a partial-least squares analysis.…”

“…In this context several source separation algorithms have been proposed to separate the vocals from the instruments and remix these components accordingly. Previous approaches used a harmonic/percussive sound separation (HPSS; Buyens et al, 2014) algorithm, nonnegative matrix factorization (NMF), multi-layer perceptrons (MLP), deep recurrent neural networks (DRNN), and convolutional autoencoders (DCAE) in order to separate different sources within an audio mixture (Pons et al, 2016;Gajȩcki and Nogueira, 2018). However, all these algorithms were implemented in non-real-time fashion to perform source separation.…”

BSS Eval or Peass? Predicting the Perception of Singing-Voice Separation

“…High-quality separation of the singing voice from accompanying instruments is an important yet difficult task serving many applications, from remixing and upmixing music [1] to increasing vocal intelligibility for the hearing impaired [2]. Unfortunately, source separation introduces distortions and artifacts, consequently degrading the sound quality of the extracted source.…”

A model for music complexity applied to music preprocessing for cochlear implants

“…Complexity reduction was achieved by extracting vocals/bass/drums from stereo recordings and attenuating the other instruments with an adjustable attenuation parameter. A similar approach to reduce music complexity by remixing the music was described in [4] and [5]. In [6] a different approach to reduce (spectral) complexity was investigated in order to increase melody clarity and ease of listening for CI users in instrumental music with reduced-rank approximations of music signals.…”