Audio Caption: Listen and Tell

Wu, Mengyue; Dinkel, Heinrich; Yu, Kai

doi:10.1109/icassp.2019.8682377

Cited by 48 publications

(48 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, two different datasets for audio captioning were presented, Audio Caption and Audio-Caps [6,7]. Audio Caption is partially released, and contains 3710 domain-specific (hospital) video clips with their audio tracks, and annotations that were originally obtained in Mandarin Chinese and afterwards translated to English using machine translation [6]. The annotators had access and viewed the videos.…”

Section: Introductionmentioning

confidence: 99%

Clotho: an Audio Captioning Dataset

Drossos

Lipping

Virtanen

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

172

127

View full text Add to dashboard Cite

Audio captioning is the novel task of general audio content description using free text. It is an intermodal translation task (not speech-to-text), where a system accepts as an input an audio signal and outputs the textual description (i.e. the caption) of that signal. In this paper we present Clotho, a dataset for audio captioning consisting of 4981 audio samples of 15 to 30 seconds duration and 24 905 captions of eight to 20 words length, and a baseline method to provide initial results. Clotho is built with focus on audio content and caption diversity, and the splits of the data are not hampering the training or evaluation of methods. All sounds are from the Freesound platform, and captions are crowdsourced using Amazon Mechanical Turk and annotators from English speaking countries. Unique words, named entities, and speech transcription are removed with post-processing. Clotho is freely available online (https://zenodo.org/record/3490684).

show abstract

Section: Introductionmentioning

confidence: 99%

Clotho: an Audio Captioning Dataset

Drossos

Lipping

Virtanen

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

172

127

View full text Add to dashboard Cite

show abstract

“…The authors of [128] presented an unsupervised image captioning framework based on a new alignment method that allows the simultaneous integration of visual and textual streams through semantic learning of multimodal embeddings of the language and vision domains. Moreover, a multimodal model can also aggregate motion information [174], acoustic information [175], temporal information [176], etc. from successive frames to assign a caption for each one.…”

Section: Image Captioningmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

“…is the task of automatically generating human-like content description of an audio signal using free text. Recent progress has been focused on the development of caption datasets [2,3,4], in which novel algorithms [5,6,7] are cultivated and fostered rapidly. However, little attention has been addressed on the automatic evaluation metrics.…”

Section: Automated Audio Captioningmentioning

confidence: 99%

Can Audio Captions Be Evaluated with Image Caption Metrics?

Zhou¹,

Zhang²,

Xu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Automated audio captioning aims at generating textual descriptions for an audio clip. To evaluate the quality of generated audio captions, previous works directly adopt image captioning metrics like SPICE and CIDEr, without justifying their suitability in this new domain, which may mislead the development of advanced models. This problem is still unstudied due to the lack of human judgment datasets on caption quality. Therefore, we firstly construct two evaluation benchmarks, AudioCaps-Eval and Clotho-Eval. They are established with pairwise comparison instead of absolute rating to achieve better inter-annotator agreement. Current metrics are found in poor correlation with human annotations on these datasets. To overcome their limitations, we propose a metric named FENSE, where we combine the strength of Sentence-BERT in capturing similarity, and a novel Error Detector to penalize erroneous sentences for robustness. On the newly established benchmarks, FENSE outperforms current metrics by 14-25% accuracy. 1

show abstract

Audio Caption: Listen and Tell

Cited by 48 publications

References 23 publications

Clotho: an Audio Captioning Dataset

Clotho: an Audio Captioning Dataset

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Can Audio Captions Be Evaluated with Image Caption Metrics?

Contact Info

Product

Resources

About