Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations

Cartwright, Mark; Seals, Ayanna; Nov, Oded; Bello, Juan Pablo

doi:10.1109/icassp.2018.8461833

Cited by 18 publications

(43 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The new label collection aimed to resolve all three of these issues. Instead of simple present/not present checkboxes, the annotators interacted with multiple timelines alongside a spectrogram representation, on which they could drag out time regions indicating the extent of each sound event (comparable to Audio Annotator [17]). Annotators quickly became adept at marking time regions on this display and we judge their timings to be precise at least to 0.1 sec resolution (based on informal spot-checks).…”

Section: Strong-labeled Datasetmentioning

confidence: 99%

The Benefit of Temporally-Strong Labels in Audio Event Classification

Hershey

Ellis

Fonseca

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

To reveal the importance of temporal precision in ground truth audio event labels, we collected precise (∼0.1 sec resolution) "strong" labels for a portion of the AudioSet dataset. We devised a temporallystrong evaluation set (including explicit negatives of varying difficulty) and a small strong-labeled training subset of 67k clips (compared to the original dataset's 1.8M clips labeled at 10 sec resolution). We show that fine-tuning with a mix of weak-and stronglylabeled data can substantially improve classifier performance, even when evaluated using only the original weak labels. For a ResNet-50 architecture, d on the strong evaluation data including explicit negatives improves from 1.13 to 1.39. The new labels are available as an update to AudioSet.

show abstract

Section: Strong-labeled Datasetmentioning

confidence: 99%

The Benefit of Temporally-Strong Labels in Audio Event Classification

Hershey

Ellis

Fonseca

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Crowdsourcing studies aimed at eliciting labels for training datasets from non-experts have also shown how differences in how information is shown [3] or what sorts of interruptions a crowdworker experiences [55] can impact the quality of labels obtained. Others have explored the value of incentivizing label and other data collection from non-experts according to their usefulness for model development (e.g., [56]).…”

Section: Knowledge Elicitation For Expert Decision Makingmentioning

confidence: 99%

“…Fields like judgment and decision making, in which eliciting (often probabilistic) prior beliefs is a topic of study, have provided evidence that how knowledge is elicited can affect the usefulness of the information. Some evidence of the effects of elicitation approaches can also be found in sub-areas of computer science that have focused mostly on non-experts like crowdsourced labeling (e.g., [3]) and active learning (e.g., [4]). However, to date few attempts have been made to characterize the space of decisions that ML researchers and practitioners make in eliciting domain knowledge from experts, and elicitation itself is rarely a topic in ML research.…”

Section: Introductionmentioning

confidence: 99%

A Survey of Domain Knowledge Elicitation in Applied Machine Learning

Kerrigan

Hullman

Bertini

2021

MTI

View full text Add to dashboard Cite

Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields.

show abstract

“…It has been reported that expert annotators do not even have to listen to audio signals, and perform the annotation task only with spectrograms [41]. However, despite the efficiency on identifying changes in the audio event, past studies have also suggested that it requires some experiences to recognize and interpret the visual patterns of spectrogram representation [12]. In the context of interactive machine learning for novice users, it is not clear whether spectrograms can be used to inspect audio contents and whether the spectrogram representation is the best way for information visualization.…”

Section: Sound Recognition and Annotationmentioning

confidence: 99%

Investigating audio data visualization for interactive sound recognition

Ishibashi

Nakao

Sugano

2020

Proceedings of the 25th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Interactive machine learning techniques have a great potential to personalize media recognition models for each individual user by letting them browse and annotate a large amount of training data. However, graphical user interfaces (GUIs) for interactive machine learning have been mainly investigated in image and text recognition scenarios, not in other data modalities such as sound. In a scenario where users browse a large amount of audio files to search and annotate target samples corresponding to their own sound recognition classes, it is difficult for them to easily navigate through the overall sample structure due to the non-visual nature of audio data. In this work, we investigate the design issue for interactive sound recognition by comparing different visualization techniques ranging from audio spectrograms to deep learning-based audio-to-image retrieval. Based on an analysis of the user study, we clarify the advantages and disadvantages of audio visualization techniques, and provide design implications for interactive sound recognition GUIs using a massive amount of audio samples. CCS CONCEPTS • Human-centered computing → Human computer interaction (HCI).

show abstract

Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations

Cited by 18 publications

References 11 publications

The Benefit of Temporally-Strong Labels in Audio Event Classification

The Benefit of Temporally-Strong Labels in Audio Event Classification

A Survey of Domain Knowledge Elicitation in Applied Machine Learning

Investigating audio data visualization for interactive sound recognition

Contact Info

Product

Resources

About