TIAR: Text-Image-Audio Retrieval with weighted multimodal re-ranking

Chi, Pei-Shan; Feng, Yong; Zhou, Mingliang; Xiong, Xiancai; Wang, Yongheng; Qiang, Baohua

doi:10.1007/s10489-023-04669-3

Cited by 3 publications

(1 citation statement)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, the single-stage task of image-speech involves using a self-supervised approach to learn large-scale image-speech pairs using visual and audio encoders separately (Li et al, 2023a;Reddy et al, 2021;Rodriguez et al, 2023;. Images and speech are embedded into a shared representation space to capture the interaction information between the two modalities (Chi et al, 2023;Chung et al, 2020;Huang et al, 2023b;Wang et al, 2024b).…”

Section: Introductionmentioning

confidence: 99%

An Efficient Channel-level Pruning for CNNs without Fine-tuning

Sun

Liu

et al. 2021

2021 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

An increasing number of Chinese people are troubled by different degrees of visual impairment, which has made the modal conversion between a single image or video frame in the visual field and the audio expressing the same information a research hotspot. Deep learning technologies such as OCR+Vocoder and Im2Wav enable English audio synthesis or image-to-sound matching in a self-supervised manner. However, the audio data used for training is limited and English is not universal for visually impaired people with different educational levels. Therefore, for the sake of solving the problems of data volume and language applicability to improve the reading efficiency of visually impaired people, a set of imageto-speech framework CLIP-KNN-Fastspeech2 based on the Chinese context was constructed. The framework integrates multiple basic models and adopts the strategy of independent pretraining and joint fine-tuning. First, the Chinese CLIP and Fastspeech2 text-to-speech models were pre-trained on two public datasets, MUGE and Baker, respectively, and their convergence was verified. Subsequently, joint fine-tuning was performed using a self-built Braille image dataset. Experimental results on multiple public datasets such as VGGSound, Flickr8k, ImageHear, and the self-built Braille dataset BIT-DP show that the model has improved objective indicators such as BLEU4,FAD(Fréchet Audio Distance), WER(Word Error Ratio), and even inference speed. This verifies that the constructed model still has the ability to synthesize high-quality speech under limited data, and also proves the effectiveness of the joint training strategy that integrates multiple basic models.

show abstract