Multimodal Data Fusion-Moving From Domain-Specific Algorithms to Transdomain Understanding for Accelerated Solution Development

Abstract

The exploding availability of new data streams now enables insights to be garnered through the integration (fusion) of multiple data sources (modalities); however, currently, it remains difficult to predict a priori which multimodal data fusion (MMDF) methods and architectures will be best suited for a novel application, leading to trial-and-error approaches that are inefficient in both time and cost. Although MMDF strategies are being applied ad hoc in many different fields (e.g., healthcare, autonomous navigation, robotics, manufacturing, transportation, etc.), results are reported in different journals using different terminology for different target audiences. These applications have diverse needs (assessment, prediction, detection, etc.), resources (number and quality of available data streams), and constraints (social, economic, and physics); however, they share many properties when these lists are evaluated at the metalevel. The authors propose that a shared MMDF framework is urgently needed to allow aggregation of insights across applications, with the end goal of developing an MMDF recommendation tool that overcomes the need for trial-and-error approaches to expedite implementation of new solutions across a wide range of disciplines. To provide context for this Special Issue and provide an overview of key related issues, this editorial presents an abbreviated summary of the current state of MMDF usage from multiple perspectives, based on input from a March 2018 workshop, and finally presents four key research priorities for the field of MMDF.

Artificial Intelligence and Laryngeal Cancer: From Screening to Prognosis: A State of the Art Review

“…Advances in the last decade contribute to the ongoing shift toward integration of multimodal data analysis, a strategy that mimics the way in which humans learn by integrating multiple data types. 15 This review is organized into 2 parts. Part 1 describes AI built on specific data modalities, highlighting their insights in LC.…”

“…Before 2010, most ML models were built only to process a single data modality. Advances in the last decade contribute to the ongoing shift toward integration of multimodal data analysis, a strategy that mimics the way in which humans learn by integrating multiple data types 15 …”

On the Use of Predictive Models for Improving the Quality of Industrial Maintenance: An Analytical Literature Review of Maintenance Strategies

“…dependency at the lowest level of features (or raw input unprocessed data), (ii) intermediate-fusion assumes a dependency at a more abstract, semantic level; and (iii) decision-based fusion assumes no dependency at all in the input, but only later at the level of decisions. The above described assumption has the following implications, as argued in [47]: (i) there are no established, standard methods to identify feature dependencies in multiple sensors and modalities; (ii) the technology exists, but there are no standard methods to extract unbiased feature from raw data, and therefore deep learning methods are preferred; (iii) there are basic techniques to handle modality fusion when dealing with missing information; (iv) it is unclear what are the relevant features to be learned, in the sense that a trial-and-error process of feature engineering is employed for the shallow ML algorithms, i.e. Decision Tree, SVM, kNN; and (v) multimodal data fusion best practices i.e.…”