A review on multimodal machine learning in medical diagnostics

Abstract

<abstract><p>Nowadays, the increasing number of medical diagnostic data and clinical data provide more complementary references for doctors to make diagnosis to patients. For example, with medical data, such as electrocardiography (ECG), machine learning algorithms can be used to identify and diagnose heart disease to reduce the workload of doctors. However, ECG data is always exposed to various kinds of noise and interference in reality, and medical diagnostics only based on one-dimensional ECG data is not trustable enough. By extracting new features from other types of medical data, we can implement enhanced recognition methods, called multimodal learning. Multimodal learning helps models to process data from a range of different sources, eliminate the requirement for training each single learning modality, and improve the robustness of models with the diversity of data. Growing number of articles in recent years have been devoted to investigating how to extract data from different sources and build accurate multimodal machine learning models, or deep learning models for medical diagnostics. This paper reviews and summarizes several recent papers that dealing with multimodal machine learning in disease detection, and identify topics for future research.</p></abstract>

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

“…The self-attention mechanism combined a deep learning encoder-decoder for the segmentation task 25 . The need to eliminate noise and distortion in data stream associated with electrocardiography (ECG) has been addressed using multimodal deep learning method which combines other data streams for improved diagnosis 30 . The fusion of data streams from several 3D neuroimaging into a pattern representing an informative latent embedding has been investigated.…”

“…Furthermore, the increasing nature of multimodality in biomedical data and the constrained information represented in a single modality are motivations for obtaining sufficient information for disease diagnosis 29 . With this, the unimodal learning is fast becoming obsolete so that the multimodal represents state-of-the-art owning to its capability to improve the robustness of models with the diversity of data 30 , and this has been widely applied to speech recognition, image processing, sentiment analysis and forensic applications. The multimodal approach has the advantage of uniformly analyzing heterogeneous features and fuses them into a common representational space.…”

Editorial: Artificial intelligence and machine learning in pediatric surgery

“…Ongoing research into computer vision algorithms, utilizing convolutional neural networks for the analysis of medical images, holds the promise of more accurate diagnoses ( 15 , 16 ). Future developments may even witness integration of sophisticated multi-modal algorithms, combining diverse data sources for highly precise predictions of specific medical conditions ( 17 ).…”