A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom

Shaik, Thanveer; Tao, Xiaohui; Li, Lin; Xie, Haoran; Velásquez, Juan D.

doi:10.1016/j.inffus.2023.102040

Cited by 32 publications

(4 citation statements)

References 146 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Privacy and ethical considerations are essential in MMLA, and recent works emphasize the importance of responsible data practices. The literature also addresses challenges related to data integration, interoperability, and the development of effective visualization techniques to make sense of multimodal data [21]. Thus, MMLA is at the forefront of educational research, offering a promising avenue for advancing LA.…”

Section: Multi-modal Learning Analytics (Mmla)mentioning

confidence: 99%

Multi-Modal LA in Personalized Education Using Deep Reinforcement Learning Based Approach

Sharif,

Uckelmann

2024

IEEE Access

View full text Add to dashboard Cite

The demand for personalized learning experiences and effective analytics in education has significantly increased. The integration of technology in education has brought about significant changes in teaching and learning practices. In the era of digital technology, the integration of education technology in the classroom has led to a change in teaching methods and learning strategies. In this paper, we introduce the KNIGHT (AI in Education at Hochschule für Technik (HFT) Stuttgart) framework, which is a holistic solution designed to tackle the complex issue of personalized education in a digital era. The paper explores the application of multimodal data integration, the novel application of deep reinforcement learning to education analytics, and the ethical consideration of privacy-preserving personalized feedback. The proposed framework's efficacy is substantiated through a case study, demonstrating its potential to revolutionize personalized education. This paper provides a comprehensive overview of the current discourse, providing valuable insights for educators, policymakers, and researchers into the multifaceted landscape of modern education, contributing to ongoing discussions and advancements in educational technology.

show abstract

Section: Multi-modal Learning Analytics (Mmla)mentioning

confidence: 99%

Multi-Modal LA in Personalized Education Using Deep Reinforcement Learning Based Approach

Sharif,

Uckelmann

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The integration of data from different modalities is on the rise in Artificial Intelligence (AI) fields, such as Machine Learning (ML) and Deep Learning (DL), known as multimodal data fusion. This research area, with applications integrating areas from natural language processing to computer vision and beyond, has driven applications in healthcare [4,5], integrating EHRs with medical images [6][7][8][9] or signals from wearable devices [10,11]. Multimodal fusion has also led to applications such as autonomous driving [12,13], environmental sciences applications combining different sensors and satellite data [14,15], as well as many other Internet of Things (IoT) applications and system improvements [16][17][18].…”

Section: Introductionmentioning

confidence: 99%

DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era

Restrepo,

Wu,

Vásquez-Venegas

et al. 2024

Preprint

View full text Add to dashboard Cite

In the big data era, integrating diverse data modalities poses significant challenges, particularly in complex fields like healthcare. This paper introduces a new process model for multimodal Data Fusion for Data Mining, integrating embeddings and the Cross-Industry Standard Process for Data Mining with the existing Data Fusion Information Group model. Our model aims to decrease computational costs, complexity, and bias while improving efficiency and reliability. We also propose "disentangled dense fusion," a novel embedding fusion method designed to optimize mutual information and facilitate dense inter-modality feature interaction, thereby minimizing redundant information.We demonstrate the model's efficacy through three use cases: predicting diabetic retinopathy using retinal images and patient metadata, domestic violence prediction employing satellite imagery, internet, and census data, and identifying clinical and demographic features from radiography images and clinical notes. The model achieved a Macro F1 score of 0.92 in diabetic retinopathy prediction, an R-squared of 0.854 and sMAPE of 24.868 in domestic violence prediction, and a macro AUC of 0.92 and 0.99 for disease prediction and sex classification, respectively, in radiological analysis. These results underscore the Data Fusion for Data Mining model's potential to significantly impact multimodal data processing, promoting its adoption in diverse, resource-constrained settings.

show abstract

“…The increase of multimodal data, which integrates disparate data formats such as text, image, or audio, requires developing sophisticated computational techniques to process and integrate these heterogeneous data types [1][2][3]. This integration, known as multimodal data fusion, leverages mainly deep learning techniques [4,5] and is critical for building systems that can interpret complex data in a manner akin to human cognition, thereby enhancing decision-making processes in clinical applications [6][7][8][9][10][11][12][13]: with fundus photos [14], Chest X-rays [15], or even public health applications using remote sensing techniques [16][17][18][19][20][21][22][23].…”

Section: Introductionmentioning

confidence: 99%

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

Restrepo,

Wu,

Cajas

et al. 2024

Preprint

View full text Add to dashboard Cite

ObjectiveLarge-scale multi-modal deep learning models and datasets have revolutionized various domains such as healthcare, underscoring the critical role of computational power. However, in resource-constrained regions like Low and Middle-Income Countries (LMICs), GPU and data access is limited, leaving many dependent solely on CPUs. To address this, we advocate leveraging vector embeddings for flexible and efficient computational methodologies, aiming to democratize multimodal deep learning across diverse contexts.Background and SignificanceOur paper investigates the computational efficiency and effectiveness of leveraging vector embeddings, extracted from single-modal foundation models and multi-modal Vision-Language Models (VLM), for multimodal deep learning in low-resource environments, particularly in health-care applications. Additionally, we propose an easy but effective inference-time method to enhance performance by further aligning image-text embeddings.Materials and MethodsBy comparing these approaches with traditional multimodal deep learning methods, we assess their impact on computational efficiency and model performance using accuracy, F1-score, inference time, training time, and memory usage across 3 medical modalities such as BRSET (ophthalmology), HAM10000 (dermatology), and SatelliteBench (public health).ResultsOur findings indicate that embeddings reduce computational demands without compromising the model’s performance, and show that our embedding alignment method improves the performance of the models in medical tasks.DiscussionThis research contributes to sustainable AI practices by optimizing computational resources in resource-constrained environments. It highlights the potential of embedding-based approaches for efficient multimodal learning.ConclusionVector embeddings democratize multimodal deep learning in LMICs, especially in healthcare. Our study showcases their effectiveness, enhancing AI adaptability in varied use cases.

show abstract

A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom

Cited by 32 publications

References 146 publications

Multi-Modal LA in Personalized Education Using Deep Reinforcement Learning Based Approach

Multi-Modal LA in Personalized Education Using Deep Reinforcement Learning Based Approach

DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

Contact Info

Product

Resources

About