Harmonic-Percussive Source Separation with Deep Neural Networks and Phase Recovery

Abstract

Harmonic/percussive source separation (HPSS) consists in separating the pitched instruments from the percussive parts in a music mixture. In this paper, we propose to apply the recently introduced Masker-Denoiser with twin networks (MaD TwinNet) system to this task. MaD TwinNet is a deep learning architecture that has reached state-of-the-art results in monaural singing voice separation. Herein, we propose to apply it to HPSS by using it to estimate the magnitude spectrogram of the percussive source. Then, we retrieve the complex-valued short-time Fourier transform of the sources by means of a phase recovery algorithm, which minimizes the reconstruction error and enforces the phase of the harmonic part to follow a sinusoidal phase model. Experiments conducted on realistic music mixtures show that this novel separation system outperforms the previous state-of-the art kernel additive model approach.

Guitar String Separation Using Non-Negative Matrix Factorization and Factor Deconvolution

“…We showed that the mask type used for HPS stage has an effect on the source estimation quality. Hence, different HPS methods, for instance, [3,5], should be investigated. We showed that the mask type at the final masking stage needs to be chosen to optimise for a given source estimation algorithm.…”

Phase-aware Harmonic/percussive Source Separation via Convex Optimization

“…Although this method considers phase information, the phase of the audio mixture is still utilized for resynthesizing the separated time domain signals. In contrast, the method presented in [19] utilizes the sinusoidal model for modifying phase. Specifically, the time-frequency mask is estimated by a deep neural network, and phases of separated signals are estimated by the specific algorithm based on the sinusoidal model [13].…”

“…However, the phase is not modified through time-frequency masking, and thus the degraded phase from the audio mixture is utilized for resynthesizing back to the time-domain. On the other hand, [19] utilizes the sinusoidal model for recovering the phase after applying a time-frequency mask obtained by a deep neural network. However, simultaneous modification of both amplitude and phase has not been presented for HPSS.…”

Phase-recovery algorithm for harmonic/percussive source separation based on observed phase information and analytic computation

“…Phase recovery, which tries to obtain a phase spectrogram that is reasonable with respect to the given amplitude, has been applied in signal enhancement [9][10][11][12][13][14], generation [15][16][17][18], and separation [19][20][21]. It has also been applied to HPSS after the estimation of the amplitude spectrograms [22,23] for improving the quality of separation. A popular phase-recovery method is temporal linear phase unwrapping (PU) [24], which recursively calculates the subsequent phase from the instantaneous frequency based on the sinusoidal model, as described in Sect.…”

“…PU is a computationally efficient method, which makes it popular in HPSS because HPSS is mainly used as preprocessing whose computation should be cheap. In HPSS, it is usually applied to the harmonic components, and the phase of the percussive components is intact or iteratively estimated using enhanced harmonic components [23].…”