Audio Captioning Transformer

Mei, Xinhao; Liu, Xubo; Huang, Qiushi; Plumbley, Mark D.; Wang, Wenwu

doi:10.48550/arxiv.2107.09817

Cited by 8 publications

(10 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1(b). As both ASR and AAC systems output word sequences, state-of-theart AAC models also follow a Transformer based encoder-decoder framework, optimized using attention loss [15,14]. Note that the CTC loss or RNN-Transducers are not particularly applicable to the AAC task, because the token sequence in a caption need not be temporally aligned with the input spectrogram frames.…”

Section: Automatic Speech Recognitionmentioning

confidence: 99%

“…Automated audio captioning (AAC) aims to provide the information of constituent audio sources and events in a structured and easily comprehensible manner i.e., a natural language description of a given audio waveform [12,13]. Recently, Transformer based encoder-decoder frameworks are being employed to model the temporal structure of audio events [14,15] AAC is an emerging research area with several applications, such as enriching the raw textual information provided by ASR during television broadcasting and video streaming. Such integration of ASR and AAC tasks can potentially improve the viewing experience of the hearing impaired.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Joint Speech Recognition and Audio Captioning

Narisetty¹,

Tsunoo²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Speech samples recorded in both indoor and outdoor environments are often contaminated with secondary audio sources. Most endto-end monaural speech recognition systems either remove these background sounds using speech enhancement or train noise-robust models. For better model interpretability and holistic understanding, we aim to bring together the growing field of automated audio captioning (AAC) and the thoroughly studied automatic speech recognition (ASR). The goal of AAC is to generate natural language descriptions of contents in audio samples. We propose several approaches for end-to-end joint modeling of ASR and AAC tasks and demonstrate their advantages over traditional approaches, which model these tasks independently. A major hurdle in evaluating our proposed approach is the lack of labeled audio datasets with both speech transcriptions and audio captions. Therefore we also create a multi-task dataset by mixing the clean speech Wall Street Journal corpus with multiple levels of background noises chosen from the AudioCaps dataset. We also perform extensive experimental evaluation and show improvements of our proposed methods as compared to existing state-of-the-art ASR and AAC methods.

show abstract

Section: Automatic Speech Recognitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Joint Speech Recognition and Audio Captioning

Narisetty¹,

Tsunoo²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We reproduce the baselines without unavailable audio clips for a fair comparison with the proposed methods. Audio Captioning Transformer (ACT) [12] has an encoder-decoder structure from Transformer [13]. The encoder block of ACT is initialized with DeiT [14] which was trained for image classification tasks and then pre-trained on an audio tagging task using AudioSet.…”

Section: Datasetmentioning

confidence: 99%

Improving Audio-Language Learning with MixGen and Multi-Level Test-Time Augmentation

Eungbeom¹,

Kim²,

Oh³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose two novel augmentation methods 1) audiolanguage MixGen (AL-MixGen) and 2) multi-level test-time augmentation (Multi-TTA) for audio-language learning. Inspired by MixGen, which is originally applied to vision-language learning, we introduce an augmentation method for the audio-language domain. We also explore the impact of test-time augmentations and present Multi-TTA which generalizes test-time augmentation over multiple layers of a deep learning model. Incorporating AL-MixGen and Multi-TTA into the baseline achieves 47.5 SPIDEr on audio captioning, which is an +18.2% over the baseline and outperforms the state-of-the-art approach with a 5x smaller model. In audio-text retrieval, the proposed methods surpass the baseline performance as well.

show abstract

“…Transformers have also been applied to some audio modeling tasks. For example, Transformerbased models have been used for audio classification (Gong et al, 2021;Verma & Berger, 2021), captioning (Mei et al, 2021), compression (Dieleman et al, 2021, speech recognition (Gulati et al, 2020), speaker separation (Subakan et al, 2021), and enhancement (Koizumi et al, 2021). Transformers have also been used for generative audio models Verma & Chafe, 2021), which in turn have enabled further tasks in music understanding (Castellon et al, 2021).…”

Section: Transformers For Sequence Modelingmentioning

confidence: 99%

MT3: Multi-Task Multitrack Music Transcription

Gardner¹,

Simon²,

Manilow³

et al. 2021

Preprint

View full text Add to dashboard Cite

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT. 1

show abstract

Audio Captioning Transformer

Cited by 8 publications

References 15 publications

Joint Speech Recognition and Audio Captioning

Joint Speech Recognition and Audio Captioning

Improving Audio-Language Learning with MixGen and Multi-Level Test-Time Augmentation

MT3: Multi-Task Multitrack Music Transcription

Contact Info

Product

Resources

About