Crowdsourced Pairwise-Comparison for Source Separation Evaluation

Cartwright, Mark; Pardo, Bryan; Mysore, Gautham J.

doi:10.1109/icassp.2018.8462153

Cited by 7 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We collect a dataset of human judgments using modern crowdsourcing tools, which have been shown to perform similarly to expert, in-lab tests [14,15]. We present a listener with two recordings, a reference x ref and perturbed signal x per , and ask if these two audio clips are exactly same or different, and record the binary response h ∈ {0, 1}.…”

Section: Data Collection Methodologymentioning

confidence: 99%

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

Manocha¹,

Finkelstein²,

Zhang³

et al. 2020

Interspeech 2020

Self Cite

View full text Add to dashboard Cite

Assessment of many audio processing tasks relies on subjective evaluation which is time-consuming and expensive. Efforts have been made to create objective metrics but existing ones correlate poorly with human judgment. In this work, we construct a differentiable metric by fitting a deep neural network on a newly collected dataset of just-noticeable differences (JND), in which humans annotate whether a pair of audio clips are identical or not. By varying the type of differences, including noise, reverb, and compression artifacts, we are able to learn a metric that is well-calibrated with human judgments. Furthermore, we evaluate this metric by training a neural network, using the metric as a loss function. We find that simply replacing an existing loss with our metric yields significant improvement in denoising as measured by subjective pairwise comparison.

show abstract

Section: Data Collection Methodologymentioning

confidence: 99%

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

Manocha¹,

Finkelstein²,

Zhang³

et al. 2020

Interspeech 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…As we do not have access to the source tracks, we cannot evaluate the separation performance using common objective evaluation metrics. Instead, we conduct a subjective evaluation on the source separation quality (Cartwright et al, 2016(Cartwright et al, , 2018 over 51 people. Some subjects are students or faculty from the University of Rochester, others are subscribers from the International Society for Music Information Retrieval (ISMIR) community.…”

Section: Subjective Evaluation On Professional a Cappella Songsmentioning

confidence: 99%

Audiovisual Singing Voice Separation

Wang

Duan

2021

Transactions of the International Society for Music Information Retrieval

View full text Add to dashboard Cite

Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques. We propose to apply the visual information corresponding to the singers' vocal activities to further improve the quality of the separated vocal signals. The video frontend model takes the input of mouth movement and fuses it into the feature embeddings of an audio-based separation framework. To facilitate the network to learn audiovisual correlation of singing activities, we add extra vocal signals irrelevant to the mouth movement to the audio mixture during training. We create two audiovisual singing performance datasets for training and evaluation, respectively, one curated from audition recordings on the Internet, and the other recorded in house. The proposed method outperforms audio-based methods in terms of separation quality on most test recordings. This advantage is especially pronounced when there are backing vocals in the accompaniment, which poses a great challenge for audio-only methods.

show abstract