COM: Contrastive Masked-attention model for incomplete multimodal learning

Shuwei, Qian,; Wang, Chongjun

doi:10.1016/j.neunet.2023.03.003

Cited by 7 publications

(2 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers can explore different fusion strategies, such as early fusion (combining raw data), late fusion (combining model outputs), or hybrid fusion (combining intermediate representations), based on the nature of data and task at hand (Figure 5C). 84 Thirdly, in some cases, certain modalities may have incomplete or missing data, i.e., modality missingness, posing difficulties for data fusion and online use of models. 85 Solutions to address modality missingness include data interpolation/ imputation, information transfer, leveraging knowledge and priors, and integrating multimodal features.…”

Section: Multimodal Data Fusion Algorithmsmentioning

confidence: 99%

Artificial intelligence for medicine: Progress, challenges, and perspectives

Huang,

Xu,

Wang

et al. 2023

TIME

View full text Add to dashboard Cite

<p>Artificial Intelligence (AI) has transformed how we live and how we think, and it will change how we practice medicine. With multimodal big data, we can develop large medical models that enables what used to unimaginable, such as early cancer detection several years in advance and effective control of virus outbreaks without imposing social burdens. The future is promising, and we are witnessing the advancement. That said, there are challenges that cannot be overlooked. For example, data generated is often isolated and difficult to integrate from both perspectives of data ownership and fusion algorithms. Additionally, existing AI models are often treated as black boxes, resulting in vague interpretation of the results. Patients also exhibit a lack of trust to AI applications, and there are insufficient regulations to protect patients�� privacy and rights. However, with the advancement of AI technologies, such as more sophisticated multimodal algorithms and federated learning, we may overcome the barriers posed by data silos. Deeper understanding of human brain and network structures can also help to unravel the mysteries of neural networks and construct more transparent yet more powerful AI models. It has become something of a trend that an increasing number of clinicians and patients will implement AI in their life and medical practice, which in turn can generate more data and improve the performance of models and networks. Last but not the least, it is crucial to monitor the practice of AI in medicine and ensure its equity, security, and responsibility.</p>

show abstract

Section: Multimodal Data Fusion Algorithmsmentioning

confidence: 99%

Artificial intelligence for medicine: Progress, challenges, and perspectives

Huang,

Xu,

Wang

et al. 2023

TIME

View full text Add to dashboard Cite

show abstract

“… 48 However, alongside the growth in the popularity of attention mechanisms within architectures such as Transformers, there has also come an exploration of how attention, with built-in support for masking certain inputs can be used to handle missing data. 49 , 50 , 51 Our framework leverages a similar approach via a cross-attention mechanism that allows us to entirely bypass data imputation and the need for complete data in model training.…”

Section: Introductionmentioning

confidence: 99%