Combining Acoustic and Semantic Similarity for Acoustic Scene Retrieval

Sert, Mustafa; Basbug, Ahmet Melih

doi:10.1109/ism46123.2019.00036

Cited by 2 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of audio signal processing, a number of tasks, such as audio event classification/detection [5], acoustic scene recognition [6,7], and audio tagging [8] have received much attention over the past few years. In the audio event detection task, the main aim is to identify (overlapping) sound events occurring in the audio clip along with their starting and ending times.…”

Section: Introductionmentioning

confidence: 99%

Audio Captioning with Composition of Acoustic and Semantic Information

Eren

Sert

2021

Int. J. Semantic Computing

Self Cite

View full text Add to dashboard Cite

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder–decoder-based models without considering semantic information. To fill this gap, we present a novel encoder–decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder–decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance.

show abstract