Transfer Learning for Improving Singing-Voice Detection in Polyphonic Instrumental Music

Abstract

Detecting singing-voice in polyphonic instrumental music is critical to music information retrieval. To train a robust vocal detector, a large dataset marked with vocal or non-vocal label at frame-level is essential. However, frame-level labeling is time-consuming and labor expensive, resulting there is little well-labeled dataset available for singing-voice detection (S-VD). Hence, we propose a data augmentation method for S-VD by transfer learning. In this study, clean speech clips with voice activity endpoints and separate instrumental music clips are artificially added together to simulate polyphonic vocals to train a vocal /non-vocal detector. Due to the different articulation and phonation between speaking and singing, the vocal detector trained with the artificial dataset does not match well with the polyphonic music which is singing vocals together with the instrumental accompaniments. To reduce this mismatch, transfer learning is used to transfer the knowledge learned from the artificial speech-plus-music training set to a small but matched polyphonic dataset, i.e., singing vocals with accompaniments. By transferring the related knowledge to make up for the lack of well-labeled training data in S-VD, the proposed data augmentation method by transfer learning can improve S-VD performance with an F-score improvement from 89.5% to 93.2%.

Rule-Embedded Network for Audio-Visual Voice Activity Detection in Live Musical Video Streams

“…As baselines, a common and typical bi-modal recurrent neural model [19] is used as A-V baseline (Base-AV ), and a CRNN [2] trained by transfer learning is used as audiobased baseline (Base-A ) to compare the performance of the AV-VAD from more perspectives.…”

“…For evaluation metrics, event-based precision (P ), recall (R ), F-score and Error rate (ER ) [21] are used. Compared with segment-based metrics used in previous studies [22,16,2], event-based metrics are more rigorous and accurate to measure the location of events. Higher P, R, F and lower ER indicate a better performance.…”

“…To process these diverse data, voice activity detection (VAD) is an essential preprocessing in detecting the presence or absence of human voice in clips. Classical applications of VAD include speaker diarization [1], music [2] and speech [3] signal processing.…”

“…The AV-VAD proposed in this paper should detect not only speech, but also the singing of anchor. Due to the different articulation and phonation between speaking and singing, the speech activity detector does not perform well with musical clips [2]. Authors in [13] attempt to detect singing voice and speech based on the same audiobased VAD model, without distinguishing whether the speech or singing voice comes from the anchor or background.…”

Attention-Based Cross-Modal Fusion for Audio-Visual Voice Activity Detection in Musical Video Streams

“…To recognize speech and singing voices in these videos, voice activity detection (VAD) is a necessary preprocessing to identify the start and end time of human voice activities. VAD has attracted many interests due to its wide applications such as speech [1,2] and music information processing [3].…”