Enhancing multimodal deep representation learning by fixed model reuse

Xie, Zhongwei; Li, Lin; Zhong, Xian; He, Yang; Zhong, Luo

doi:10.1007/s11042-018-6556-6

Cited by 3 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…erefore, an information retrieval method based on hash transformation appears. is method is based on the paired sample pairs of different modal data, learns the corresponding hash transformation, maps the corresponding modal data features to the Hamming binary space, and then realizes faster cross-modal retrieval in this space [21]. e premise of hash transformation is that the hash codes of similar samples are also similar.…”

Section: Cross-modal Retrieval Methods Based On Hashmentioning

confidence: 99%

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Yin

Chen

2022

Scientific Programming

View full text Add to dashboard Cite

In view of the complexity of the multimodal environment and the existing shallow network structure that cannot achieve high-precision image and text retrieval, a cross-modal image and text retrieval method combining efficient feature extraction and interactive learning convolutional autoencoder (CAE) is proposed. First, the residual network convolution kernel is improved by incorporating two-dimensional principal component analysis (2DPCA) to extract image features and extracting text features through long short-term memory (LSTM) and word vectors to efficiently extract graphic features. Then, based on interactive learning CAE, cross-modal retrieval of images and text is realized. Among them, the image and text features are respectively input to the two input terminals of the dual-modal CAE, and the image-text relationship model is obtained through the interactive learning of the middle layer to realize the image-text retrieval. Finally, based on Flickr30K, MSCOCO, and Pascal VOC 2007 datasets, the proposed method is experimentally demonstrated. The results show that the proposed method can complete accurate image retrieval and text retrieval. Moreover, the mean average precision (MAP) has reached more than 0.3, the area of precision-recall rate (PR) curves are better than other comparison methods, and they are applicable.

show abstract

Section: Cross-modal Retrieval Methods Based On Hashmentioning

confidence: 99%

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Yin

Chen

2022

Scientific Programming

View full text Add to dashboard Cite

show abstract

“…Penelitian ini bertujuan untuk menangani konsep yang lebih jelas dan spesifik untuk hype dan pesan iklan penipuan tersebut. Untuk tujuan ini, bahasa iklan yang tidak jujur diklasifikasikan menjadi tiga yaitu, iklan fiktif, iklan menggertak (bluffing), dan iklan penipuan (Hou et al, 2018;Xie et al, 2019). Kategori yang berbeda dan terdefinisi dengan baik dan karakteristik masing-masing kategori serta kemungkinan implikasi dari pesan iklan terselubung telah dipelajari dalam pragmatik neo-Gricean.…”

Section: A Pendahuluanunclassified

“…Untuk itu, perusahaan menjauh dari citra iklan produk sederhana yang menginformasikan keunikan dan keunggulan produknya, dan membuat iklan berorientasi konsumen dari iklan citra perusahaan atau iklan berorientasi produsen yang mengembalikan keuntungan perusahaan kepada masyarakat. Ini adalah tren yang berkembang menjadi dua arah yang benar dengan sekaligus memperluas jangkauan area komunikasi dengan konsumen (Del Saz-Rubio, 2019;Xie et al, 2019). Sementara itu, linguistik periklanan sebagian besar berfokus pada pendekatan linguistik murni seperti linguistik teks dan semiotika, yaitu penelitian tentang bahasa iklan, dan aspek etika, yang merupakan pendekatan pragmatika/dialogistik untuk periklanan, relatif diabaikan.…”

Section: Tabel 7 Analisis Aspek Berlebihan Dan Kebohongan Pada Iklanunclassified

Analisis Multimodal pada Teks Iklan untuk Mengidentifikasi Aspek Ketulusan Iklan

et al. 2023

View full text Add to dashboard Cite

This study aims to analyze the sincerity aspect of Indonesian and Korean advertisements through multimodal analysis. This study uses a content analysis research method using the FCB Grid Model on several products, namely fast food, tobacco, and telecommunications. The selection of ad samples was carried out based on the FCB Grid Model. The results of the study show that some products such as tobacco and fast food are included in the category of products that involve low emotions and thoughts when consumers consider these products. However, telecommunications products fall into the category of products that involve high rational thinking skills, thus requiring serious consideration from consumers. Other findings show that there are several semantic and pragmatic expressions that make consumers confused in choosing a product. The most frequent violations found in products are false expressions and explanations that lead consumers astray. In addition, other violations are expression omissions, exaggerated expressions, fictitious, and omission of important information, thus making consumers feel lost in choosing products.

show abstract

“…There has been significant research and progress on the crossmodal retrieval between texts and images in the last two decades, which derives from the keyword-based text-image cross-modal retrieval [31,45] and evolves into the sentence-based [9,26,35,44,49] and even long text (paragraph) based text-image cross-modal retrieval tasks [2,4,12,24,34,41,53]. As the text-image crossmodal retrieval tasks are widely applied in different scenarios and the text in the tasks grows longer and more complicated [34,47], the event-dense character is gradually emerging, i.e., the long text might contain the descriptions of multiple events. For example, as shown in Figure 1, the instructional descriptions in the recipe text contain several events about how to handle the ingredients.…”

Section: Introductionmentioning

confidence: 99%

Cross-Modal Retrieval between Event-Dense Text and Image

Xie

Zhong

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

Self Cite

View full text Add to dashboard Cite

This paper presents a novel approach to the problem of event-dense text and image cross-modal retrieval where the text contains the descriptions of numerous events. It is known that modality alignment is crucial for retrieval performance. However, due to the lack of event sequence information in the image, it is challenging to perform the fine-grain alignment of the event-dense text with the image. Our proposed approach incorporates the event-oriented features to enhance the cross-modal alignment, and applies the event-dense text-image retrieval to the food domain for empirical validation. Specifically, we capture the significance of each event by Transformer, and combine it with the identified key event elements, to enhance the discriminative ability of the learned text embedding that summarizes all the events. Next, we produce the image embedding by combining the event tag jointly shared by the text and image with the visual embedding of the event-related image regions, which describes the eventual consequence of all the events and facilitates the event-based cross-modal alignment. Finally, we integrate text embedding and image embedding with the loss optimization empowered with the event tag by iteratively regulating the joint embedding learning for cross-modal retrieval. Extensive experiments demonstrate that our proposed event-oriented modality alignment approach significantly outperforms the state-of-the-art approach with 23.3% improvement on top-1 Recall for image-torecipe retrieval on Recipe1M 10k test set. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval.

show abstract

Enhancing multimodal deep representation learning by fixed model reuse

Cited by 3 publications

References 33 publications

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Analisis Multimodal pada Teks Iklan untuk Mengidentifikasi Aspek Ketulusan Iklan

Cross-Modal Retrieval between Event-Dense Text and Image

Contact Info

Product

Resources

About