Fast and Effective Retrieval for Large Multimedia Collections

Abstract

The indexing and retrieval of multimedia content is generally implemented by employing feature graphs. These graphs typically contain a significant number of nodes and edges to reflect the level of detail in feature detection. A higher level of detail increases the effectiveness of the results, but also leads to more complex graph structures. However, graph traversal-based algorithms for similarity are quite inefficient and computationally expensive, especially for large data structures. To deliver fast and effective retrieval especially for large multimedia collections and multimedia big data, an efficient similarity algorithm for large graphs in particular is desirable. Hence, in this paper, we define a graph projection into a 2D space (Graph Code) and the corresponding algorithms for indexing and retrieval. We show that calculations in this space can be performed more efficiently than graph traversals due to the simpler processing model and the high level of parallelization. As a consequence, we demonstrate experimentally that the effectiveness of retrieval also increases substantially, as the Graph Code facilitates more levels of detail in feature fusion. These levels of detail also support an increased trust prediction, particularly for fused social media content. In our mathematical model, we define a metric triple for the Graph Code, which also enhances the ranked result representations. Thus, Graph Codes provide a significant increase in efficiency and effectiveness, especially for multimedia indexing and retrieval, and can be applied to images, videos, text and social media information.

Explainable Multimedia Feature Fusion for Medical Applications

“…However, even with these extensions and annotations, the pure graph-based structure of an MMFG remains. As we showed in [2], this graph-based structure leads to exponential processing time of some MMIR algorithms. Particularly, for medical applications, it is important to integrate a high number of features from various sources into such an MMFG and to increase the LOD to the best possible extent.…”

“…In many cases, these are stored in the form of feature graphs [17,18]. In previous work [2,3], we presented a unifying framework-the Generic Multimedia Analysis Framework (GMAF)-and data structurethe Multimedia Feature Graph (MMFG)-to represent such multimedia features in extended detail. The GMAF utilizes selected existing technologies as plugins to support various multimedia feature detection algorithms for text (e.g., social media posts, descriptions, tag lines) [19][20][21], images (especially object detection and spatial relationships including the use of machine learning) [18][19][20]22,23], audio (transcribed to text) [22,24], and video, including metadata [25] and detected features [26,27].…”

“…Graph Codes [2,33] are a 2D projection of a multimedia feature graph based on adjacency matrix operations [34], where the feature vocabulary terms of a feature graph are presented by row and column indices, and matrix positions represent relationships between these features. It has been demonstrated that Graph Codes are efficient for similarity and recommendation searches, as they basically use linear growing matrix operations instead of exponentially growing graph-traversal-based algorithms.…”

“…In previous work on Multimedia Information Retrieval (MMIR), we demonstrated a highly efficient and effective approach for multimedia feature extraction, the semantic representation, annotation, and fusion of these features, and introduced information retrieval metrics including explainability of multimedia feature graphs [1][2][3]. Based on this MMIR research, further extensions and refinements are possible in the area of medical applications.…”

Towards Automated Semantic Explainability of Multimedia Feature Graphs

“…• A set of Semantic Feature Vocabulary Terms SFVT SMMFG and a set of Semantic Relationship Vocabulary Terms FRVT SMMFG . In previous work[7,43], we already defined the set FVT MMFG as the representation of all syntactic MMFG vocabulary terms (i.e., the textual labels of detected features). In an SMMFG, the semantic of each syntactic vocabulary term, f vt i ∈ FVT MMFG , is now represented by a semantic feature vocabulary term s f vt i ∈ SFVT SMMFG .…”