Automated audio captioning with recurrent neural networks

Abstract

We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.

Clotho: an Audio Captioning Dataset

“…The encoder is a series of bi-directional gated recurrent units (bi-GRUs) [11], similarly to [4]. The output dimensionality for the GRU layers (forward and backward GRUs have same dimensionality) is {256, 256, 256}.…”

“…Baseline method and evaluationIn order to provide an example of how to employ Clotho and some initial (baseline) results, we use a previously utilized method for audio captioning[4] which is based on an encoder-decoder scheme with attention. The method accepts as an input a length-T sequence of 64 log mel-band energies X ∈ R T ×64 , which is used as an input to a DNN which outputs a probability distribution of words.…”

“…The method accepts as an input a length-T sequence of 64 log mel-band energies X ∈ R T ×64 , which is used as an input to a DNN which outputs a probability distribution of words. The generated caption is constructed from the output of the DNN, as in[4]. We optimize the parameters of the method using the development split of Clotho and we evaluate it using the evaluation and the testing splits, separately.We first extract 64 log mel-band energies, using a Hamming window of 46 ms, with 50% overlap.…”

Polyphonic Sound Event Detection Based on Residual Convolutional Recurrent Neural Network With Semi-Supervised Loss Function

“…For example, acoustic monitoring could detect physical events, such as glass breaking, a gun firing, tires skidding, or a car crashing. SED can also be incorporated into audio captioning for understanding social media content in more detail [2], audio monitoring in smart cities [3], life assistance and healthcare [4], etc.…”

Audio Caption: Listen and Tell

“…To achieve human-like perception, using natural language to describe images( [6,7,8]) and videos has attracted much attention( [9,10,11]). Yet only little research has been made regarding audio scenes [12], which we think is due to the difference between visual Heinrich Dinkel is the co-first author. Kai Yu and Mengyue Wu are the corresponding authors.…”