Abstract:Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution to this, where any missing modalities are synthesized from the existin… Show more
“…Extensive experiments have been conducted using arbitrary single modality as input and synthesizing the rest modalities. Compared to very recent studies on multimodal MR image synthesis, [26][27][28][29][30]32,38,49 our proposed method achieves higher synthetic accuracy attribute to our advanced network architecture. Meanwhile, our proposed method paves a way in which multimodal MR image can be synthesized from only one single modality input through taking in the modality labels as extra information.…”
Section: Discussionmentioning
confidence: 81%
“…Regressions either linear or nonlinear were commonly adopted for cross‐modality MR image synthesis in early intensity transformation‐based methods. As the rapid growth of applying deep learning in MRI, 19–21 recently, deep learning‐based end‐to‐end frameworks have been investigated for multimodal MR image synthesis 13,22–32 . Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs) 33 .…”
Section: Introductionmentioning
confidence: 99%
“…As the rapid growth of applying deep learning in MRI, [19][20][21] recently, deep learning-based end-to-end frameworks have been investigated for multimodal MR image synthesis. 13,[22][23][24][25][26][27][28][29][30][31][32] Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs). 33 These deep neural network-based methods can be grouped into three main categories depending on their input/output modalities: (a) single-input single-output (SISO), (b) multi-input singleoutput (MISO), (c) multi-input multi-output (MIMO).…”
Purpose
Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis.
Methods
A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE).
Results
The proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively.
Conclusions
We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.
“…Extensive experiments have been conducted using arbitrary single modality as input and synthesizing the rest modalities. Compared to very recent studies on multimodal MR image synthesis, [26][27][28][29][30]32,38,49 our proposed method achieves higher synthetic accuracy attribute to our advanced network architecture. Meanwhile, our proposed method paves a way in which multimodal MR image can be synthesized from only one single modality input through taking in the modality labels as extra information.…”
Section: Discussionmentioning
confidence: 81%
“…Regressions either linear or nonlinear were commonly adopted for cross‐modality MR image synthesis in early intensity transformation‐based methods. As the rapid growth of applying deep learning in MRI, 19–21 recently, deep learning‐based end‐to‐end frameworks have been investigated for multimodal MR image synthesis 13,22–32 . Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs) 33 .…”
Section: Introductionmentioning
confidence: 99%
“…As the rapid growth of applying deep learning in MRI, [19][20][21] recently, deep learning-based end-to-end frameworks have been investigated for multimodal MR image synthesis. 13,[22][23][24][25][26][27][28][29][30][31][32] Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs). 33 These deep neural network-based methods can be grouped into three main categories depending on their input/output modalities: (a) single-input single-output (SISO), (b) multi-input singleoutput (MISO), (c) multi-input multi-output (MIMO).…”
Purpose
Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis.
Methods
A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE).
Results
The proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively.
Conclusions
We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.
The coronavirus disease, called COVID-19, which is spreading fast worldwide since the end of 2019, and has become a global challenging pandemic. Until 27th May 2020, it caused more than 5.6 million individuals infected throughout the world and resulted in greater than 348,145 deaths. CT images-based classification technique has been tried to use the identification of COVID-19 with CT imaging by hospitals, which aims to minimize the possibility of virus transmission and alleviate the burden of clinicians and radiologists. Early diagnosis of COVID-19, which not only prevents the disease from spreading further but allows more reasonable allocation of limited medical resources. Therefore, CT images play an essential role in identifying cases of COVID-19 that are in great need of intensive clinical care. Unfortunately, the current public health emergency, which has caused great difficulties in collecting a large set of precise data for training neural networks. To tackle this challenge, our first thought is transfer learning, which is a technique that aims to transfer the knowledge from one or more source tasks to a target task when the latter has fewer training data. Since the training data is relatively limited, so a transfer learning-based DensNet-121 approach for the identification of COVID-19 is established. The proposed method is inspired by the precious work of predecessors such as CheXNet for identifying common Pneumonia, which was trained using the large Chest X-ray14 dataset, and the dataset contains 112,120 frontal chest X-rays of 14 different chest diseases (including Pneumonia) that are individually labeled and achieved good performance. Therefore, CheXNet as the pre-trained network was used for the target task (COVID-19 classification) by fine-tuning the network weights on the small-sized dataset in the target task. Finally, we evaluated our proposed method on the COVID-19-CT dataset. Experimentally, our method achieves state-of-the-art performance for the accuracy (ACC) and F1-score. The quantitative indicators show that the proposed method only uses a GPU can reach the best performance, up to 0.87 and 0.86, respectively, compared with some widely used and recent deep learning methods, which are helpful for COVID-19 diagnosis and patient triage. The codes used in this manuscript are publicly available on GitHub at (
https://github.com/lichun0503/CT-Classification
).
“…Attention U-Net [15] highlights the foreground via the supplement of more semantic information in the encoder parts. Hi-Net [16] utilizes more information from different modalities via the fusion of each learned feature representations. Liu et al [17] present a sample balancing strategy via the assignment different weights to the edge and background pixels to further improve the extraction accuracy.…”
Textured surface anomaly detection is a significant task in industrial scenarios. In order to further improve the detection performance, we proposed a novel two-stage approach with an attention mechanism. Firstly, in the segmentation network, the feature extraction and anomaly attention modules are designed to capture the detail information as much as possible and focus on the anomalies, respectively. To strike dynamic balances between these two parts, an adaptive scheme where learnable parameters are gradually optimized is introduced. Subsequently, the weights of the segmentation network are frozen, and the outputs are fed into the classification network, which is trained independently in this stage. Finally, we evaluate the proposed approach on DAGM 2007 dataset which consists of diverse textured surfaces with weakly-labeled anomalies, and the experiments demonstrate that our method can achieve 100% detection rates in terms of TPR (True Positive Rate) and TNR (True Negative Rate).
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
