Signal Processing Methods for Music Transcription

Animal vocal communication is a broad and multi-disciplinary field of research. Studying various aspects of communication can provide key elements for understanding animal behavior, evolution, and cognition. Given the large amount of acoustic data accumulated from automated recorders, for which manual annotation and analysis is impractical, there is a growing need to develop algorithms and automatic methods for analyzing and identifying animal sounds. In this study we developed an automatic detection and analysis system based on audio signal processing algorithms and deep learning that is capable of processing and analyzing large volumes of data without human bias. We selected the White Spectacled Bulbul (Pycnonotus xanthopygos) as our bird model because it has a complex vocal communication system with a large repertoire which is used by both sexes, year-round. It is a common, widespread passerine in Israel, which is relatively easy to locate and record in a broad range of habitats. Like many passerines, the Bulbul’s vocal communication consists of two primary hierarchies of utterances, syllables and words. To extract each of these units’ characteristics, the fundamental frequency contour was modeled using a low degree Legendre polynomial, enabling it to capture the different patterns of variation from different vocalizations, so that each pattern could be effectively expressed using very few coefficients. In addition, a mel-spectrogram was computed for each unit, and several features were extracted both in the time-domain (e.g., zero-crossing rate and energy) and frequency-domain (e.g., spectral centroid and spectral flatness). We applied both linear and non-linear dimensionality reduction algorithms on feature vectors and validated the findings that were obtained manually, namely by listening and examining the spectrograms visually. Using these algorithms, we show that the Bulbul has a complex vocabulary of more than 30 words, that there are multiple syllables that are combined in different words, and that a particular syllable can appear in several words. Using our system, researchers will be able to analyze hundreds of hours of audio recordings, to obtain objective evaluation of repertoires, and to identify different vocal units and distinguish between them, thus gaining a broad perspective on bird vocal communication.

show abstract

“…Spectral Bandwidth , as defined in Klapuri and Davy (2007) , which is a weighted standard deviation of the spectrum in a given audio segment:…”

Section: Methodsmentioning

confidence: 99%

Identification, Analysis and Characterization of Base Units of Bird Vocal Communication: The White Spectacled Bulbul (Pycnonotus xanthopygos) as a Case Study

Marck

Vortman

Kolodny

et al. 2022

Front. Behav. Neurosci.

View full text Add to dashboard Cite

show abstract

“…As a result, 98% of the pairs of stimuli presented in the four-sounds BWS trials had a difference equal to or greater than the spectral centroid JND. The spectral centroid of each sound was computed and averaged for each sound with the Librosa library (Klapuri and Davy, 2007). The loudness of each sound sample was equalized following the European Broadcast Union (EBU) norm on loudness (R-128) with the ffmpeg library (Python Package Index).…”

Section: Stimulimentioning

confidence: 99%

Best-worst scaling, an alternative method to assess perceptual sound qualities

Rosi

Ravillion

Houix

et al. 2022

JASA Express Letters

View full text Add to dashboard Cite

When designing sound evaluation experiments, researchers rely on listening test methods, such as rating scales (RS). This work aims to investigate the suitability of best-worst scaling (BWS) for the perceptual evaluation of sound qualities. To do so, 20 participants rated the “brightness” of a corpus of instrumental sounds (N = 100) with RS and BWS methods. The results show that BWS procedure is the fastest and that RS and BWS are equivalent in terms of performance. Interestingly, participants preferred BWS over RS. Therefore, BWS is an alternative method that reliably measures perceptual sound qualities and could be used in many-sounds paradigm.

show abstract

“…Audio feature extraction was performed using the Python module "librosa" (McFee et al, 2015). The features were: Mel-frequency cepstral coeffi-cients (MFCCs), which are widely used features for characterising and detecting voice signals (Klapuri and Davy, 2006); several spectral features like spectral centroids (Klapuri and Davy, 2006), spectral bandwidth (Klapuri and Davy, 2006), spectral roll-off (McFee et al, 2015) and spectral contrast (Jiang et al, 2002); and a 12-bit chroma vector (McFee et al, 2015). For each sentence, we used the timestamp to clip the audio file with a buffer of ±2 s to ensure the full audio of the utterance was captured.…”

Section: Argumentative Relation Classificationmentioning

confidence: 99%

M-Arg: Multimodal Argument Mining Dataset for Political Debates with Audio and Transcripts

Mestre¹,

Milicin²,

Middleton³

et al. 2021

Proceedings of the 8th Workshop on Argument Mining

View full text Add to dashboard Cite

Argumentation mining aims at extracting, analysing and modelling people's arguments, but large, high-quality annotated datasets are limited, and no multimodal datasets exist for this task. In this paper, we present M-Arg, a multimodal argument mining dataset with a corpus of US 2020 presidential debates, annotated through crowd-sourced annotations. This dataset allows models to be trained to extract arguments from natural dialogue such as debates using information like the intonation and rhythm of the speaker. Our dataset contains 7 hours of annotated US presidential debates, 6527 utterances and 4104 relation labels, and we report results from different baseline models, namely a text-only model, an audio-only model and multimodal models that extract features from both text and audio. With accuracy reaching 0.86 in multimodal models, we find that audio features provide added value with respect to text-only models.

show abstract

Signal Processing Methods for Music Transcription

Cited by 257 publications

References 336 publications

Identification, Analysis and Characterization of Base Units of Bird Vocal Communication: The White Spectacled Bulbul (Pycnonotus xanthopygos) as a Case Study

Identification, Analysis and Characterization of Base Units of Bird Vocal Communication: The White Spectacled Bulbul (Pycnonotus xanthopygos) as a Case Study

Best-worst scaling, an alternative method to assess perceptual sound qualities

M-Arg: Multimodal Argument Mining Dataset for Political Debates with Audio and Transcripts

Contact Info

Product

Resources

About