Learning Sound Event Classifiers from Web Audio with Noisy Labels

Abstract

As sound event classification moves towards larger datasets, issues of label noise become inevitable. Web sites can supply large volumes of user-contributed audio and metadata, but inferring labels from this metadata introduces errors due to unreliable inputs, and limitations in the mapping. There is, however, little research into the impact of these errors. To foster the investigation of label noise in sound event classification we present FSDnoisy18k, a dataset containing 42.5 hours of audio across 20 sound classes, including a small amount of manually-labeled data and a larger quantity of realworld noisy data. We characterize the label noise empirically, and provide a CNN baseline system. Experiments suggest that training with large amounts of noisy data can outperform training with smaller amounts of carefully-labeled data. We also show that noiserobust loss functions can be effective in improving performance in presence of corrupted labels.

Learning With Out-of-Distribution Data for Audio Classification

“…In contrast, only 10 % of the training set is verified: this is the data that is used to train the auxiliary classifier. It has been estimated [2] that 45 % of the unverified labels are incorrect, and that 84 % of the incorrect labels are OOD.…”

“…However, due to the uncontrolled/miscellaneous nature of these sources of data, irrelevant (OOD) instances are likely to be encountered when curating the data. For example, Freesound Annotator [6] is a platform of datasets comprised of over 260 K audio samples annotated by the public, where the authors of this platform have observed a considerable number of OOD instances [2]. OOD corruption can occur for a number of reasons, such as uncertainty in the sound (e.g.…”

Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers

“…The evaluation of the model-agnostic methods is conducted using the FSDnoisy18k dataset [5], an open dataset containing 42.5 hours of audio across 20 sound event classes, including a small amount of manually-labeled data and a larger quantity of real-world noisy data. The audio content is taken from Freesound [23], and the dataset was curated using the Freesound Annotator [24].…”

“…In contrast, the mean absolute error equation (MAE) treats every instance equally (hence avoiding the weighting aspect), and it has been shown theoretically that MAE can be used as a loss function robust against label noise [22]. Nevertheless, MAE has been reported to cause other difficulties in training that lead to performance drop [5,14]. To take advantage of the benefits of CCE (its weighting property) and MAE (its noiserobustness), a generalization of those functions has been recently proposed in [14].…”

“…where q controls the closeness of Lq to CCE or MAE. Lq is the top performing noise-robust loss function among several evaluated for sound event classification [5]. Further details about Lq can be found in [14].…”