Entity-aware Image Caption Generation

Lu, Di; Whitehead, Spencer; Huang, Lifu; Ji, Heng; Chang, Shu‐Fen

doi:10.18653/v1/d18-1435

Cited by 54 publications

(32 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So leveraging textual information from the news at the article level or even the sentence level is not fine-grained enough for finding appropriate named entities. Secondly, to overcome the out-of-vocabulary (OOV) problem caused by named entities, previous works [3,12] follow a two-stage framework. They first generate the 'template caption' which contains the entity types (namely 'placeholders').…”

Section: Introductionmentioning

confidence: 99%

ICECAP: Information Concentrated Entity-aware Image Captioning

Chen

Jin

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Most current image captioning systems focus on describing general image content, and lack background knowledge to deeply understand the image, such as exact named entities or concrete events. In this work, we focus on the entity-aware news image captioning task which aims to generate informative captions by leveraging the associated news articles to provide background knowledge about the target image. However, due to the length of news articles, previous works only employ news articles at the coarse article or sentence level, which are not fine-grained enough to refine relevant events and choose named entities accurately. To overcome these limitations, we propose an Information Concentrated Entity-aware news image CAPtioning (ICECAP) model, which progressively concentrates on relevant textual information within the corresponding news article from the sentence level to the word level. Our model first creates coarse concentration on relevant sentences using a cross-modality retrieval model and then generates captions by further concentrating on relevant words within the sentences. Extensive experiments on both BreakingNews and GoodNews datasets demonstrate the effectiveness of our proposed method, which outperforms other state-of-the-arts. CCS CONCEPTS • Computing methodologies → Natural language generation; Computer vision.

show abstract

Section: Introductionmentioning

confidence: 99%

ICECAP: Information Concentrated Entity-aware Image Captioning

Chen

Jin

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Thus, every representation from knowledge graphs must not be ambiguous regarding how they are presented, demanding consistency. This property is what allows knowledge graphs to be a reliable form of representation when representing large amounts of data-thus making them a viable option when trying to embed certain domains or documents in fields such as natural language processing [14], multimodal processing [15], etc. Inspired by this, the proposed model utilizes knowledge graph embeddings for the extraction of global relations from documents.…”

Section: Knowledge Graph Embeddingmentioning

confidence: 99%

GREG: A Global Level Relation Extraction with Knowledge Graph Embedding

et al. 2020

View full text Add to dashboard Cite

In an age overflowing with information, the task of converting unstructured data into structured data are a vital task of great need. Currently, most relation extraction modules are more focused on the extraction of local mention-level relations-usually from short volumes of text. However, in most cases, the most vital and important relations are those that are described in length and detail. In this research, we propose GREG: A Global level Relation Extractor model using knowledge graph embeddings for document-level inputs. The model uses vector representations of mention-level 'local' relation's to construct knowledge graphs that can represent the input document. The knowledge graph is then used to predict global level relations from documents or large bodies of text. The proposed model is largely divided into two modules which are synchronized during their training. Thus, each of the model's modules is designed to deal with local relations and global relations separately. This allows the model to avoid the problem of struggling against loss of information due to too much information crunched into smaller sized representations when attempting global level relation extraction. Through evaluation, we have shown that the proposed model yields high performances in both predicting global level relations and local level relations consistently.

show abstract

“…Knowledge-driven Generation. Deep Neural Networks have been applied to generate natural language to describe structured knowledge bases (Duma and Klein, 2013;Konstas and Lapata, 2013;Flanigan et al, 2016;Hardy and Vlachos, 2018;Pourdamghani et al, 2016;Trisedya et al, 2018;Xu et al, 2018;Madotto et al, 2018;Nie et al, 2018), biographies based on attributes (Lebret et al, 2016;Chisholm et al, 2017;Kaffee et al, 2018;Wang et al, 2018a;Wiseman et al, 2018), and image/video captions based on background entities and events (Krishnamoorthy et al, 2013;Lu et al, 2018). To handle unknown words, we design an architecture similar to pointer-generator networks (See et al, 2017) and copy mechanism (Gu et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

PaperRobot: Incremental Draft Generation of Scientific Ideas

Wang¹,

Huang²,

Jiang³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We present a PaperRobot who performs as an automatic research assistant by (1) conducting deep understanding of a large collection of human-written papers in a target domain and constructing comprehensive background knowledge graphs (KGs); (2) creating new ideas by predicting links from the background KGs, by combining graph attention and contextual text attention; (3) incrementally writing some key elements of a new paper based on memory-attention networks: from the input title along with predicted related entities to generate a paper abstract, from the abstract to generate conclusion and future work, and finally from future work to generate a title for a follow-on paper. Turing Tests, where a biomedical domain expert is asked to compare a system output and a human-authored string, show PaperRobot generated abstracts, conclusion and future work sections, and new titles are chosen over human-written ones up to 30%, 24% and 12% of the time, respectively. 1

show abstract

Entity-aware Image Caption Generation

Cited by 54 publications

References 27 publications

ICECAP: Information Concentrated Entity-aware Image Captioning

ICECAP: Information Concentrated Entity-aware Image Captioning

GREG: A Global Level Relation Extraction with Knowledge Graph Embedding

PaperRobot: Incremental Draft Generation of Scientific Ideas

Contact Info

Product

Resources

About