A deep grouping fusion neural network for multimedia content understanding

Abstract

How Deep Neural Networks (DNNs) best cope with the understanding of multimedia contents still remains an open problem, mainly due to two factors. First, conventional DNNs cannot effectively learn the representations of the images with sparse visual information. For example, the images describing knowledge concepts in textbooks. Second, existing DNNs cannot effectively capture the fine‐grained interactions between the images and text descriptions. To address these issues, we propose a deep Cross‐Media Grouping Fusion Network (CMGFN), which mainly has two distinctive properties: 1) CMGFN can effectively learn visual features from the images with sparse visual information. This is achieved by first progressively adjusting the attention of convolution filters to valuable visual regions, and then enhancing the use of key visual information in feature construction. 2) By a cross‐media grouping co‐attention mechanism, CMGFN can effectively use the interactions between visual features of different semantics and textual descriptions, to learn cross‐media features representing different fine‐grained semantics in different groups. Empirical studies demonstrate that CMGFN not only achieves state‐of‐the‐art performance on the multimedia documents containing sparse visual information, but also shows superior general applicability on other multimedia data, e.g., the multimedia fake news.

What’s On the Menu? Towards Predicting Nutritional Quality of Food Environments

“…(b) Food category prediction model. We embed Edamam training data containing menu items concatenated with their ingredients using a pre-trained MPNet model [49]. We then cluster the training data using HDBSCAN, which we treat as a ground truth food category.…”

“…We first utilize a range of data to create specialized language models for extracting features [48]. Next, we use a state-of-the-art sentence embedding model, MPNet [49] to embed menu item names with their ingredients to create ingredient-contextualized food clusters. These clusters act as food category pseudo-labels to train a model that maps menu item names (without their ingredients) to the learned pseudo-labels.…”

“…To group the food items, we take both food item names and ingredients in the Edamam dataset by converting these data into a sentence "Food Name made with query of ingredients", for example, "Egg scrambled made with butter, egg, salt, pepper." We then embed these texts using a pre-trained MPNET model [49], map it to a lower dimension space, and finally cluster these embeddings to create food category pseudo-labels for each food item.…”

GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification Task

“…Additionally, as mentioned earlier, we generated 3 more training sets featuring the same tweets but with new labels obtained through 3 distinct prompting techniques utilizing GPT-4. Subsequently, we fine-tuned eight transformer-based LLMs, namely Bert [31], Albert [32], Deberta [33], BerTweet [34], MPNet [35], and three Robertabased models pre-trained on i) a general Twitter dataset (TRob) [36], ii) a Twitter sentiment dataset (TRobSen) [23], and iii) a Twitter stance dataset (TRobStan) [23]. These models were separately fine-tuned using our four training datasets.…”

A Dense Retrieval System and Evaluation Dataset for Scientific Computational Notebooks

“…Neural retrieval models are then developed for semantic retrieval [10]. Among them, dense retrieval methods [11] based on largescale pre-trained language models [12], [13], e.g., DPR [14] and SBERT [15], map both query and document into a continuous vector space (dense vector space) where semantically similar words, phrases and sentences are closer to each other, and thus also called semantic retrieval methods. Similarly, two types of code modeling methods exist: Information Retrieval (IR)-based and Machine Learning (ML)-based methods.…”