While music information retrieval (MIR) has made substantial progress in automatic analysis of audio similarity for Western music, it remains unclear whether these algorithms can be meaningfully applied to cross-cultural analyses of more diverse samples. Here we collected perceptual ratings from 62 participants using a global sample of 30 traditional songs, and compared these ratings against both pre-existing expert annotations and state-of-the-art audio similarity algorithms. We found that different methods of perceptual ratings all produced similar, moderate levels of inter-rater reliability comparable to previous studies, but that agreement between human and automated methods was always low regardless of the specific methods used to calculate musical similarity. Our findings suggest that current MIR methods are unable to measure cross-cultural music similarity in perceptually meaningful ways. We propose future directions to enable meaningful automatic analysis of all the world’s music.

Although MIR has demonstrated great success in automatic analysis of Western music, no study has tested automatic algorithms against perceptual ground-truth data for a global musical sample. It thus remains unknown whether MIR algorithms can be meaningfully applied to automatically compare diverse music from around the world. In this pilot study, we aim to establish ground truth perceptual data on similarity between diverse musical recordings from across the world and use this perceptual data to test the accuracy of existing audio similarity algorithms. Preliminary results (two participants, ten recordings) suggest that perceptual ratings of musical similarity are significantly correlated between participants, but these similarities are only weakly correlated with similarities measured by an automatic algorithm. While this is consistent with more pessimistic assessments of MIR's current ability to accommodate non-Western music, we hypothesize that collecting more perceptual data, comparing against more algorithms, and creating new algorithms based on more universal musical theories will enable meaningful automatic analysis of all the world's music within the near future. This would have important implications for our understanding of cross-cultural music diversity, including applications to domains such as music recommendation and cultural heritage preservation.1.