Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval

Zeng, Donghuo; Wang, Yanan; Wu, Jianming; Ikeda, Kazushi

doi:10.1109/ism55400.2022.00007

Cited by 2 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works on audio-visual retrieval tasks exploit supervised representation learning methods to generate new features across modalities in a common space [13], [14], [15], [16], [39], [40], [41], [42], such that the audio-visual features can be measured directly. Inspired by the C-CCA [39] that aims at finding linear transformations for each modality, C-DCCA [40] tries to learn non-linear features in the common space by using deep learning methods.…”

Section: ) Audio-visual Retrieval Taskmentioning

confidence: 99%

“…Inspired by the C-CCA [39] that aims at finding linear transformations for each modality, C-DCCA [40] tries to learn non-linear features in the common space by using deep learning methods. Deep learning methods by using rank loss to optimize the predicted distances, such as TNN-C-CCA [13], and CCTL [16] models, which apply triplet losses as the objective functions to achieve better results than other CCA-variant methods. The EICS model [42] learns two different common spaces to capture modality-common and modality-specific features, which achieves the SOTA results so far.…”

Section: ) Audio-visual Retrieval Taskmentioning

confidence: 99%

“…In addition to the binary accuracy reported by most of the previous works, we evaluate the 7-class accuracy as did the SOTA method [12] to eliminate the effect of the data imbalance. For the audio-visual retrieval task, we apply the mean average precision (mAP) as previous works [16], [42] to evaluate our model.…”

Section: B Evaluation Metricmentioning

confidence: 99%

“…transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion [8], [9]. It has been broadly applied to multimodal tasks such as video-level sentiment analysis [10], [11], [12], and audio/text-video retrieval tasks [13], [14], [15], [16].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

et al. 2023

Self Cite

View full text Add to dashboard Cite

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss-first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit-second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the stateof-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning. INDEX TERMSMultimodal transfer learning, knowledge distillation, fundamental model. YANAN WANG (Student Member, IEEE) received the B.S. degree in engineering from Aoyama Gakuin University, in 2015, and the M.S. degree in engineering from The University of Electro-Communications, Japan, in 2017. He is currently pursuing the Ph.D. degree in engineering with Keio University, Japan. He is an Associate Research Engineer in multimodal modeling topics with KDDI Research Inc. His research interests include multimodal representation learning, emotion recognition, knowledge graph, and graph representation learning. He is a Student Member of JSAI. He is a regular member of IEICE and the Editorial Committee of IEICE Human Communication Group.

show abstract