Explainable Multimedia Feature Fusion for Medical Applications

Abstract

Due to the exponential growth of medical information in the form of, e.g., text, images, Electrocardiograms (ECGs), X-ray, multimedia, etc., the management of a patient’s data has become a huge challenge. Particularly, the extraction of features from various different formats and their representation in a homogeneous way are areas of particular interest in medical applications. Multimedia Information Retrieval (MMIR) frameworks, like the Generic Multimedia Analysis Framework (GMAF), can contribute to solving this problem, when adapted to special requirements and modalities of medical applications. In this paper, we demonstrate how typical multimedia processing techniques can be extended and adapted to medical applications and how these applications benefit from employing a Multimedia Feature Graph (MMFG) and specialized, highly efficient indexing structures in the form of Graph Codes. These Graph Codes are transformed to feature relevant Graph Codes by employing a modified Term Frequency Inverse Document Frequency (TFIDF) algorithm, which further supports value ranges and Boolean operations required in the medical context. On this basis, various metrics for the calculation of similarity, recommendations, and automated inferencing and reasoning can be applied supporting the field of diagnostics. Finally, the presentation of these new facilities in the form of explainability is introduced and demonstrated. Thus, in this paper, we show how Graph Codes contribute new querying options for diagnosis and how Explainable Graph Codes can help to quickly understand medical multimedia formats.

Parallelization Strategies for Graph-Code-Based Similarity Search

“…The extracted features are contributed to the MMFG, which can be further processed. Extensions of MMFGs have led to semantic analysis, such as Semantic Multimedia Feature Graphs (SMMFGs) and Explainable SMMFGs (ESMMFGs) [24]. Despite these extensions, the graph-based structure of MMFGs remains and can lead to slow processing times.…”

“…With the introduction of semantics to the MMFGs in [24], we introduced additional metrics to improve the efficiency and effectiveness of Graph Codes for MMIR. First, the feature discrimination M DIS is defined as the difference in the number of nonzero Graph Code fields for two feature vocabulary terms of a given Graph Code or Semantic Graph Code.…”

“…This method has been used in similar cases [55]. Another approach is to employ the previously described Feature Relevance [24]. Graph Codes with only relevant features should increase the performance with a minimal loss of accuracy.…”

Smart Multimedia Information Retrieval

“…"Smart MMIR" thus describes expressive, scalable, interoperable, explainable and human understandable MMIR solutions. In previous work [2][3][4], we already introduced, defined, and evaluated the core components, which contribute to Smart MMIR. However, the interoperability of these components and a corresponding formal model is a foundation for further improvements in the problem areas, which were mentioned above.…”

“…In [3], we further showed that not only feature graphs, but also the indexing structures, such as, for example, graph codes, can be automatically transformed into humanunderstandable texts. Based on this, further metrics for semantic graph codes were introduced [4] as follows:…”