Combining visual and acoustic features for audio classification tasks

Nanni, Loris; Costa, Yandre M. G.; Lucio, Diego Rafael; Silla, Carlos N.; Brahnam, Sheryl

doi:10.1016/j.patrec.2017.01.013

Cited by 74 publications

(63 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The spectrogram has rich sound features [12][13][14]. To enhance the system capabilities, we considered introducing CNN [15], which deals with images.…”

Section: Cnnmentioning

confidence: 99%

“…As this study focuses on sound analysis, we investigated recent studies on sound signal processing using AI techniques, and then discovered several studies on audio feature analysis using a spectrogram. A spectrogram image obtained by short-time Fourier transform contains rich information regarding sound characteristics [12][13][14]; for this, a convolution neural network (CNN) [15] is employed to classify the sound from the inputted spectrogram. Justin and Juan [16] addressed the classification of an environmental sound using CNN, into which a spectrogram-like image (mel-spectrogram) is inputted.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Snack Texture Estimation System Using a Simple Equipment and Neural Network Model

et al. 2019

View full text Add to dashboard Cite

Texture evaluation is manually performed in general, and such analytical tasks can get cumbersome. In this regard, a neural network model is employed in this study. This paper describes a system that can estimate the food texture of snacks. The system comprises a simple equipment unit and an artificial neural network model. The equipment simultaneously examines the load and sound when a snack is pressed. The neural network model analyzes the load change and sound signals and then outputs a numerical value within the range (0,1) to express the level of textures such as “crunchiness” and “crispness”. Experimental results validate the model’s capacity to output moderate texture values of the snacks. In addition, we applied the convolutional neural network (CNN) model to classify snacks and the capability of the CNN model for texture estimation is discussed.

show abstract

“…The spectrogram has rich sound features [12][13][14]. To enhance the system capabilities, we considered introducing CNN [15], which deals with images.…”

Section: Cnnmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Snack Texture Estimation System Using a Simple Equipment and Neural Network Model

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Most ML approaches in animal call classification take their lead from automated speech recognition by virtue of the commonalities between human speech and birdcalls. These ML approaches include supervised neural networks (including deep learning neural networks) [17]- [21], unsupervised neural networks [22], support vector machines [23]- [25], decision trees [26], [27], random forests [28], [29], and hidden markov model [30]- [34]. Despite the significant amount of research into the automated classification of birdcalls, there is not yet an adequate method for field recordings due to the challenges associated with birdcall classification, such as the high variability in calls.…”

Section: A Birdcalls In Acoustic Recordingmentioning

confidence: 99%

Data-Efficient Classification of Birdcall through Convolutional Neural Networks Transfer Learning

Efremova¹,

Sankupellay

Konovalov

2019

2019 Digital Image Computing: Techniques and Applications (DICTA)

View full text Add to dashboard Cite

Deep learning Convolutional Neural Network (CNN) models are powerful classification models but require a large amount of training data. In niche domains such as bird acoustics, it is expensive and difficult to obtain a large number of training samples. One method of classifying data with a limited number of training samples is to employ transfer learning. In this research, we evaluated the effectiveness of birdcall classification using transfer learning from a larger base dataset (2814 samples in 46 classes) to a smaller target dataset (351 samples in 10 classes) using the ResNet-50 CNN. We obtained 79% average validation accuracy on the target dataset in 5-fold cross-validation. The methodology of transfer learning from an ImageNet-trained CNN to a project-specific and a much smaller set of classes and images was extended to the domain of spectrogram images, where the base dataset effectively played the role of the ImageNet.

show abstract

“…Sound classification and recognition has been included among the pattern recognition tasks for different application domains, e.g. speech recognition [1], music classification [2], environmental sound recognition or biometric identification [3]. In the traditional pattern recognition framework (preprocessing, feature extraction and classification) features have generally been extracted from the actual audio traces (e.g.…”

Section: Introductionmentioning

confidence: 99%

Data augmentation approaches for improving animal audio classification

Nanni

Maguolo

Paci

2020

Ecological Informatics

127

View full text Add to dashboard Cite

In this paper we present ensembles of classifiers for automated animal audio classification, exploiting different data augmentation techniques for training Convolutional Neural Networks (CNNs). The specific animal audio classification problems are i) birds and ii) cat sounds, whose datasets are freely available. We train five different CNNs on the original datasets and on their versions augmented by four augmentation protocols, working on the raw audio signals or their representations as spectrograms. We compared our best approaches with the state of the art, showing that we obtain the best recognition rate on the same datasets, without ad hoc parameter optimization.Our study shows that different CNNs can be trained for the purpose of animal audio classification and that their fusion works better than the stand-alone classifiers. To the best of our knowledge this is the largest study on data augmentation for CNNs in animal audio classification audio datasets using the same set of classifiers and parameters. Our MATLAB code is available at https://github.com/LorisNanni .

show abstract

Combining visual and acoustic features for audio classification tasks

Cited by 74 publications

References 14 publications

Snack Texture Estimation System Using a Simple Equipment and Neural Network Model

Snack Texture Estimation System Using a Simple Equipment and Neural Network Model

Data-Efficient Classification of Birdcall through Convolutional Neural Networks Transfer Learning

Data augmentation approaches for improving animal audio classification

Contact Info

Product

Resources

About