ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition

Abstract

Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations obtained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text representations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their respective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose Image-text Alignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer-based pretrained textual embeddings can be better utilized. ITA first locally and globally aligns regional object tags and image-level captions as visual contexts, concatenates them with the input texts as a new cross-modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction between the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practical and robust to noises from images. In our experiments, we show that ITA models can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets, even without image information.

A Multimodal Named Entity Recognition Model for Power Equipment Based on Deep Neural Network

“…Therefore, the performance of named entity recognition models in practical data is not ideal. In recent years, many studies on multimodal named entity recognition have incorporated text corresponding images as supplementary information into text fusion [12][13][14][15] to improve the problem of information accuracy. However, these studies did not pay attention to the large amount of noise generated by irrelevant image information.…”

Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge

“…They employ diverse cross-modal attention mechanisms to facilitate the interaction between text and images. Recently, Wang et al (2021a) points out that the performance limitations of such methods are largely attributed to the disparities in distribution between different modalities. Despite Wang et al (2022c) try to mitigate the aforementioned issues by using further refining cross-modal attention, training this end-to-end cross-modal Transformer architectures imposes significant demands on computational resources.…”

“…Despite Wang et al (2022c) try to mitigate the aforementioned issues by using further refining cross-modal attention, training this end-to-end cross-modal Transformer architectures imposes significant demands on computational resources. Due to the aforementioned limitations, ITA (Wang et al, 2021a) and MoRe (Wang et al, 2022a) attempt to use a new paradigm to address MNER. ITA circumvents the challenge of multi-modal alignment by forsaking the utilization of raw visual features and opting for OCR and image captioning techniques to convey image information.…”

“…The version of ChatGPT used in experiments is gpt-3.5-turbo and sampling temperature is set to 0. For a fair comparison, PGIM chooses to use the same text encoder XLM-RoBERTa large (Conneau et al, 2019) as ITA (Wang et al, 2021a), PromptM-NER (Wang et al, 2022b), CAT-MNER (Wang et al, 2022c) and MoRe (Wang et al, 2022a).…”

“…In addition, recent studies (Wang et al, 2021b; has shown that introducing additional document-level context on the basis of the original text can significantly improve the performance of NER models. Therefore, recent studies (Wang et al, 2021a(Wang et al, , 2022a aim to solve the MNER task using the Text-Text (T+T) paradigm. In these approaches, images are reasonably converted into textual representations through techniques such as image caption and optical character recognition (OCR).…”