Seeing Sound

Abstract

Audio annotation is key to developing machine-listening systems; yet, effective ways to accurately and rapidly obtain crowdsourced audio annotations is understudied. In this work, we seek to quantify the reliability/redundancy trade-off in crowdsourced soundscape annotation, investigate how visualizations affect accuracy and efficiency, and characterize how performance varies as a function of audio characteristics. Using a controlled experiment, we varied sound visualizations and the complexity of soundscapes presented to human annotators. Results show that more complex audio scenes result in lower annotator agreement, and spectrogram visualizations are superior in producing higher quality annotations at lower cost of time and human labor. We also found recall is more affected than precision by soundscape complexity, and mistakes can be often attributed to certain sound event characteristics. These findings have implications not only for how we should design annotation tasks and interfaces for audio data, but also how we train and evaluate machine-listening systems.

Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing

“…DHH participants in prior work liked waveforms while recording samples in a lab setting [8]; we explore their value for samples recorded in daily life. Spectrograms show the frequency spectrum over time, are often used for scientifc analyses (e.g., bioacoustics [16]), and can be difcult to interpret for novice hearing users [12,35]. Early work showed frequency information was inadequate for DHH users in a sound identifcation task [54]; we briefy explore DHH participants' opinions of spectrograms for displaying sound activity.…”

Automatic Segmentation of Ethnomusicological Field Recordings

“…Five second long excerpts were extracted from recordings in these collections. To manually label them into the target classes, we enhanced the web-based audio annotator tool [15], so that it can be controlled exclusively by the keyboard. This makes labelling very fast when an excerpt contains just one class (e.g., speech).…”

Describing the sounds of nature: Using onomatopoeia to classify bird calls for citizen science

“…As such, finding methods that both engage citizen scientists and allow the swift and accurate categorisation of complex vocalisations will provide an advantage over the output of a much smaller number of experts labouring alone, while also advancing ecological science in the public sphere. Some of the method for acoustic annotation being explored include the pairing of short snippets of sound with visualisations such as spectrograms [3,8], or just providing visualisations [9]. While this presents an advantage for representing certain species' calls (e.g.…”

“…However, many forms of machine learning rely upon human intelligence to provide the pre-labelled datasets that they are trained upon, and the production of these annotated datasets is a time-consuming process [ 2 ]. While crowd-sourced human intelligence offers a potential solution, issues remain in terms of participant accuracy and efficiency [ 3 , 4 ], as well as how to motivate continued involvement in the task. However, citizen science offers additional benefits to ecological projects, such as engaging the public with scientific processes [ 5 ] and conservation agendas [ 6 , 7 ].…”

“…Some of the method for acoustic annotation being explored include the pairing of short snippets of sound with visualisations such as spectrograms [ 3 , 8 ], or just providing visualisations [ 9 ]. While this presents an advantage for representing certain species’ calls (e.g.…”