Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform

Pölsterl, Sebastian; Wolf, Tom Nuno; Wachinger, Christian

doi:10.1007/978-3-030-87240-3_66

Cited by 27 publications

(9 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The above attention-based fusion methods rescaled features through complementary information from another modality, while Pölsterl et al [52] proposed a dynamic affine transform module that shifted the feature map. The proposed modules dynamically produced scale factor and offset conditional on both image and clinical data.…”

Section: Attention-based Fusion Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

et al. 2023

View full text Add to dashboard Cite

The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various image (e.g., radiological, pathological and camera image) and non-image data (e.g., clinical data andgenomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multi-modal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multi-modal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (1) overview of current multi-modal learning workflows, (2) summarization of multi-modal fusion methods, (3) discussion of the performance, (4) applications in disease diagnosis and prognosis, and (5) challenges and future directions.

show abstract

Section: Attention-based Fusion Methodsmentioning

confidence: 99%

“…The missing value is a common problem for some structured data. The ones with a high missing rate were usually discarded directly, while the other missing data were imputed with the average value, mode value, or values of similar samples selected by K-nearest neighbors [16,24], and some works added missing status as features [52,68].…”

Section: Structured Datamentioning

confidence: 99%

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Second, applying the recently introduced Dynamic Affine Feature Map Transform (DAFT, Fig. 3 ) [ 19 ]. It is a general-purpose module for CNNs that incites or represses high-level concepts learned from a 3D image by conditioning feature maps of a convolutional layer on both a patient's image and tabular clinical information.…”

Section: Methodsmentioning

confidence: 99%

Deep learning for the prediction of type 2 diabetes mellitus from neck-to-knee Dixon MRI in the UK biobank

Wachinger,

Wolf,

Pölsterl

2023

Heliyon

View full text Add to dashboard Cite

“…Each landmark patch was then fed into the CNN models, which produced the final classification result using the maximum voting strategy. Pölsterl et al [35] proposed the dynamic affine feature map transform, an auxiliary module for CNNs that dynamically incites or represses each feature map of a convolutional layer based on both image and tabular biomarkers. A more detailed overview of deep learning algorithms for Alzheimer's disease classification can be found in [36], [14], [37].…”

Section: Literature Reviewmentioning

confidence: 99%

A Multi-Stream Convolutional Neural Network for Classification of Progressive MCI in Alzheimer’s Disease Using Structural MRI Images

Ashtari-Majlan

Seifi

Dehshibi

2022

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

Early diagnosis of Alzheimer's disease and its prodromal stage, also known as mild cognitive impairment (MCI), is critical since some patients with progressive MCI will develop the disease. We propose a multi-stream deep convolutional neural network fed with patch-based imaging data to classify stable MCI and progressive MCI. First, we compare MRI images of Alzheimer's disease with cognitively normal subjects to identify distinct anatomical landmarks using a multivariate statistical test. These landmarks are then used to extract patches that are fed into the proposed multi-stream convolutional neural network to classify MRI images. Next, we train the architecture in a separate scenario using samples from Alzheimer's disease images, which are anatomically similar to the progressive MCI ones and cognitively normal images to compensate for the lack of progressive MCI training data. Finally, we transfer the trained model weights to the proposed architecture in order to fine-tune the model using progressive MCI and stable MCI data. Experimental results on the ADNI-1 dataset indicate that our method outperforms existing methods for MCI classification, with an F1-score of 85.96%.

show abstract

Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform

Cited by 27 publications

References 25 publications

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Deep learning for the prediction of type 2 diabetes mellitus from neck-to-knee Dixon MRI in the UK biobank

A Multi-Stream Convolutional Neural Network for Classification of Progressive MCI in Alzheimer’s Disease Using Structural MRI Images

Contact Info

Product

Resources

About