Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information

Ye, Zhongjie; Wang, Helin; Yang, Dongchao; Zou, Yuexian

doi:10.48550/arxiv.2110.06100

Cited by 2 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mei et al [19] proposed a full Transformer-based audio captioning method to improve the capability of modelling global and fine-grained temporal information. Ye et al [20] proposed a fully supervised audio captioning model based on the multi-modal attention module, which utilizes acoustic and semantic information to generate captions. Xu et al [21] pre-trained the audio encoder on textaudio retrieval tasks, enhancing the representation capability of the audio encoder for audio captioning.…”

Section: B Fully Supervised Audio Captioningmentioning

confidence: 99%

“…Fully Supervised Audio Captioning: We compare our method with fully supervised audio captioning methods: ACT [19], MAAC [20], Xu et al [21], Prefix AAC [22], RLSSR [23], RECAP [24], and ACTUAL [7]. All of which are open source and not trained with additional data.…”

Section: Baselinesmentioning

confidence: 99%

“…58.4 ± 0.21 16.9 ± 0.30 38.5 ± 0.30 41.6 ± 0.46 17.8 ± 0.08 12.1 ± 0.14 26.9 ± 0.26 MAAC[20] 57.0 ± 0.53 16.0 ± 0 40. 37.7 ± 0.37 41.3 ± 0.56 17.7 ± 0.22 12.3 ± 0.13 26.8 ± 0.32 Xu et al [21] 56.9 ± 0.15 16.0 ± 0.39 37.9 ± 0.33 41.8 ± 0.69 17.9 ± 0.15 12.7 ± 0.07 27.3 ± 0.34 WSAC [29] 54.5 ± 0.05 12.6 ± 0.14 35.9 ± 0.04 35.7 ± 0.33 16.9 ± 0.02 11.8 ± 0.01 23.8 ± 0.17 Ours 56.4 ± 0.24 15.6 ± 0.22 37.5 ± 0.17 40.3 ± 0.47 17.3 ± 0.17 11.9 ± 0.19 26.1 ± 0.27 † We use the original results listed in the paper since these works include results for in-domain and cross-domain scenarios.…”

mentioning

confidence: 99%

See 2 more Smart Citations

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

Zhuang

Huang

et al. 2019

Mathematical Biosciences and Engineering

View full text Add to dashboard Cite

The crystallization kinetics and melting behavior of nylon 10,10 in neat nylon 10,10 and in nylon 10,10 -montmorillonite (MMT) nanocomposites were systematically investigated by differential scanning calorimetry. The crystallization kinetics results show that the addition of MMT facilitated the crystallization of nylon 10,10 as a heterophase nucleating agent; however, when the content of MMT was high, the physical hindrance of MMT layers to the motion of nylon 10,10 chains retarded the crystallization of nylon 10,10, which was also confirmed by polarized optical microscopy. However, both nylon 10,10 and nylon 10,10 -MMT nanocomposites exhibited multiple melting be-havior under isothermal and nonisothermal crystallization conditions. The temperature of the lower melting peak (peak I) was independent of MMT content and almost remained constant; however, the temperature of the highest melting peak (peak II) decreased with increasing MMT content due to the physical hindrance of MMT layers to the motion of nylon 10,10 chains.

show abstract

Section: B Fully Supervised Audio Captioningmentioning

confidence: 99%

Section: Baselinesmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

Zhuang

Huang

et al. 2019

Mathematical Biosciences and Engineering

View full text Add to dashboard Cite

show abstract

“…CLIP-AAC [27] adds a text encoder to the conventional encoder-decoder architecture for contrastive learning. MAAC [6] employs an LSTM-based multimodal attention decoder to incorporate both the acoustic and the semantic information. P-LocalAFT [7] employs CNN-10 from PANN [28] as the encoder and an local information assisted attention-free transformer as the decoder.…”

Section: Setupmentioning

confidence: 99%

“…In order to address this challenge, prior works have extensively used the transfer learning framework. Some works employed audio tagging or sound event detection as the pretraining task for the audio encoder [5,6,7,8,9]. Others utilized pretrained language models such as GPT-2 [10,11,12,13] and BART [14,15] as the text decoder to enhance the semantic quality of the captions.…”

Section: Introductionmentioning

confidence: 99%

Republic of Korea (South Korea)

Kim¹

The Crime of Aggression

View full text Add to dashboard Cite

Despite recent progress in semantic image synthesis, complete control over image style remains a challenging problem. Existing methods require reference images to feed style information into semantic layouts, which indicates that the style is constrained by the given image. In this paper, we propose a model named RUCGAN for user controllable semantic image synthesis, which utilizes a singular color to represent the style of a specific semantic region. The proposed network achieves reference-free semantic image synthesis by injecting color as userdesired styles into each semantic layout, and is able to synthesize semantic images with unusual colors. Extensive experimental results on various challenging datasets show that the proposed method outperforms existing methods, and we further provide an interactive UI to demonstrate the advantage of our approach for style controllability. The codes and UI are available at: https://github.com/BenjaminJonghyun/RUCGAN

show abstract

Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information

Cited by 2 publications

References 14 publications

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

Republic of Korea (South Korea)

Contact Info

Product

Resources

About