M3Care: Learning with Missing Modalities in Multimodal Healthcare Data

Abstract

Multimodal electronic health record (EHR) data are widely used in clinical applications. Conventional methods usually assume that each sample (patient) is associated with the unified observed modalities, and all modalities are available for each sample. However, missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. Existing methods mostly rely on solving a generative model that learns a mapping from the latent space to the original input space, which is an unstable ill-posed inverse problem. To relieve the underdetermined system, we propose a model solving a direct problem, dubbed learning with Missing Modalities in Multimodal healthcare data (M 3 Care). M 3 Care is an end-to-end model compensating the missing information of the patients with missing modalities to perform clinical analysis. Instead of generating raw missing data, M 3 Care imputes the task-related information of the missing modalities in the latent space by the auxiliary information from each patient's similar neighbors, measured by a task-guided modality-adaptive similarity metric, and thence conducts the clinical tasks. The taskguided modality-adaptive similarity metric utilizes the uncensored modalities of the patient and the other patients who also have the * Both authors contributed equally to this research.

Graph-Based Patient Representation for Multimodal Clinical Data: Addressing Data Heterogeneity

“…To compare the predictive performance of our system, we plan to compare with three other baselines: i) HAIM [Soenksen et al, 2022]: uses late-fusion of modality-specific representations, ii) M3Care [Zhang et al, 2022]: missing modality imputation in latent space using similar patients, and iii) ARMOUR [Liu et al, 2023]: missing modality imputation using transformer based model.…”

“…Yet, these models may introduce noise by assuming an ambiguous latent space for clinical data [Zhang et al, 2022]. Zhang et al [2022] propose identifying similar patients based on available modalities to impute missing modalities in the latent space directly, addressing cases where modalities are missing due to system errors or noise. However, it overlooks systematic biases in patient data, contributing to measurement heterogeneity, such as the absence of X-ray for patients who may not require it for their diagnosis.…”

“…One stream of studies uses entire data or set of patients represented as graphs and these representions are used for downstream tasks. For example, Zhang et al [2022] find the similar patients by representing patients by their profiles and use for further downstream task of missing modality imputation. A common approach of application of GNN to clinical context is creating a large heterogeneous network with relations between patients and their associated information, such as medications or diagnosis.…”

“…However, these methods often cannot handle missing modalities. Although some research addresses data imputation within modalities, these methods do not apply to missing modalities [Zhang et al, 2022]. As emphasized in our example and D’Hondt et al [2022], these challenges mirror real-world scenarios and are significant barriers to clinicians’ adoption of CDSS.…”

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

“…(Hayat, Geras, and Shamout 2022) utilized an LSTM layer to generate a representative vector for general cases. (Zhang et al 2022) proposed to impute in the latent space with auxiliary information. These methods either require prior knowledge or assume different modalities to be similar.…”