Understanding the Message of Images with Knowledge Base Traversals

Weiland, Lydia; Hulpuș, Ioana; Ponzetto, Simone Paolo; Dietz, Laura

doi:10.1145/2970398.2970414

Cited by 4 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TagMe is a service for identifying short phrases ( spots ) in a text that can be linked to a pertinent Wikipedia page 78 . TagMe is used for text contextualization and understanding applications in English, German, and Italian 79‐82 . The method augments a plain text by identifying “anchors,” that is, portions of the text that point to Wikipedia pages related to their meanings.…”

Section: Methodsmentioning

confidence: 99%

NLPHub: An e‐Infrastructure‐based text mining hub

Coro

Panichi

Pagano

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary Text mining involves a set of processes that analyze text to extract high‐quality information. Among its large number of applications, there are experiments that tackle big data challenges using complex system architectures. However, text mining approaches are neither easy to discover and use nor easily combinable by end‐users. Furthermore, they should be contextualized within new approaches to science (eg, Open Science) that ensure longevity and reuse of methods and results. This article presents NLPHub, a distributed system that orchestrates and combines several state‐of‐the‐art text mining services that recognize spatiotemporal events, keywords, and a large set of named entities. NLPHub adopts an Open Science approach, which fosters the reproducibility, repeatability, and reusability of methods and results, by using an e‐Infrastructure supporting data‐intensive Science. NLPHub adds Open Science‐compliance to the connected services through the use of representational standards for services and computations. It also manages heterogeneous service access policies and enables collaboration and sharing facilities. This article reports a performance assessment based on an annotated corpus of named entities, which demonstrates that NLPHub can improve the performance of the single‐integrated processes by cleverly combining their output.

show abstract

Section: Methodsmentioning

confidence: 99%

NLPHub: An e‐Infrastructure‐based text mining hub

Coro

Panichi

Pagano

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…To the best of our knowledge there is no public dataset providing gist annotations for image-caption pairs. 5 Consequently, we make use of our experiments of the dataset for understanding the message of images covering the topic of non-literal and literal image-caption pairs introduced in our previous work [83].…”

Section: Methodsmentioning

confidence: 99%

Knowledge-rich image gist understanding beyond literal meaning

Weiland

Hulpuș

Ponzetto

et al. 2018

Data & Knowledge Engineering

Self Cite

View full text Add to dashboard Cite

We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their literal meaning, our work addresses the identification of connotations, i.e., iconic meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. Our proposed algorithm brings together aspects of entity linking and clustering, subgraph selection, semantic relatedness, and learning-to-rank in a novel way. In addition to this novel task and a complete evaluation of our approach, we introduce a novel dataset to foster further research on this problem. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process. However, we also show experimentally that stateof-the-art image and text understanding is better at dealing with literal meanings of image-caption pairs, with non-literal pairs being instead generally more difficult to detect, thus paving the way for future work on understanding the message of images beyond their literal content.

show abstract

“…The main idea of the approach is to represent the images and their affiliated text by concepts from a knowledge base and conduct a query-specific knowledge graph generation, which in the case of [3] represents the gist and which in our case is used to create a mapping between query image and target class. In the knowledge base representation of Wikipedia we are using, article pages and categories are the concepts.…”

Section: Methodsmentioning

confidence: 99%

“…Ranking the Nodes. To conduct the ranking of concepts best representing the image-text pair, we use 16 different features, e.g., graph connectivity and content based measures, boolean and features based on the article texts of Wikipedia concepts (for further details, please refer to [3]). As there is no gold standard about the relevance of the so far collected concepts given (recall: a concept in the knowledge base has an equivalent node in the knowledge graph, but as we are not doing a graph ranking, we switch back the notation of concept), we rely on a pre-trained model using the data and gold standard of [3]to rank the concepts.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Understanding the Gist of Images - Ranking of Concepts for Multimedia Indexing

Weiland,

Ponzetto,

Effelsberg

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Nowadays, where multimedia data is continuously generated, stored, and distributed, multimedia indexing, with its purpose of grouping similar data, becomes more important than ever. Understanding the gist (=message) of multimedia instances is framed in related work as a ranking of concepts from a knowledge base, i.e., Wikipedia. We cast the task of multimedia indexing as a gist understanding problem. Our pipeline benefits from external knowledge and two subsequent learningto-rank (l2r) settings. The first l2r produces a ranking of concepts representing the respective multimedia instance. The second l2r produces a mapping between the concept representation of an instance and the targeted class topic(s) for the multimedia indexing task. The evaluation on an established big size corpus (MIRFlickr25k, with 25,000 images), shows that multimedia indexing benefits from understanding the gist. Finally, with a MAP of 61.42, it can be shown that the multimedia indexing task benefits from understanding the gist. Thus, the presented end-to-end setting outperforms DBM and competes with Hashing-based methods.

show abstract

Understanding the Message of Images with Knowledge Base Traversals

Cited by 4 publications

References 36 publications

NLPHub: An e‐Infrastructure‐based text mining hub

NLPHub: An e‐Infrastructure‐based text mining hub

Knowledge-rich image gist understanding beyond literal meaning

Understanding the Gist of Images - Ranking of Concepts for Multimedia Indexing

Contact Info

Product

Resources

About