X-Norm: Exchanging Normalization Parameters for Bimodal Fusion

Yin, Yufeng; Xu, Jiashu; Zu, Tianxin; Soleymani, Mohammad

doi:10.1145/3536221.3556581

Cited by 2 publications

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multimodal representation learning has been increasingly important as modern intelligent applications require a comprehensive understanding of vision, language and speech (Yin et al, 2022). To learn meaningful latent spaces, unsupervised alignment between different modality inputs has been proven effective as an additional layer of structural information about the data.…”

Section: Multimodal Alignmentmentioning

confidence: 99%

ALCAP: Alignment-Augmented Music Captioner

He,

Hao,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Music captioning has gained significant attention in the wake of the rising prominence of streaming media platforms. Traditional approaches often prioritize either the audio or lyrics aspect of the music, inadvertently ignoring the intricate interplay between the two. However, a comprehensive understanding of music necessitates the integration of both these elements. In this study, we delve into this overlooked realm by introducing a method to systematically learn multimodal alignment between audio and lyrics through contrastive learning. This not only recognizes and emphasizes the synergy between audio and lyrics but also paves the way for models to achieve deeper cross-modal coherence, thereby producing high-quality captions. We provide both theoretical and empirical results demonstrating the advantage of the proposed method, which achieves new state-of-the-art on two music captioning datasets. Our code is publicly available at https://github.com/ zihaohe123/ALCAP.

show abstract

Section: Multimodal Alignmentmentioning

confidence: 99%