Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency

Multimodal news analytics using measures of cross-modal entity and context consistency

“…Moreover, the utilized event classification approach [31] distinguishes between event types such as football, elections, or types of natural disasters rather than between sub-types or concrete event instances such as UEFA Champions League or 2020 U.S. elections. Despite these limitations the results are superior to the scene classification approach used in our previous work [30], as discussed in more detail in Sect. 5.5.…”

“…Event Features: In our initial approach [30], we used a more general image descriptor for scene classification to extract features for events since related approaches for event classification [1,2,43] have not considered many event types that are relevant for news. Recently, we have presented a dataset and ontology-driven deep learning approach for event classification [31].…”

“…The visual event features f E are extracted from the last pooling layer of the ResNet-50 architecture [14,15]. A comparison to the previous approach [30] is conducted in Sect. 5.5.…”

“…Dataset Version 2: Please note that we have noticed a minor problem in the first version of our dataset [30] that affected circa 5% of the linked entities. As a consequence, the results slightly differ from the first version.…”

“…In this respect, appropriate computer vision approaches serve as generalized feature extractors. Unlike the more general model for scene/place classification used in [30], we utilize a novel ontology-driven deep learning approach [31] to generate features for event verification. Finally, novel measures for different entity types (persons, locations, events) as well 1 https://www.politifact.com/.…”