Distribution Matching Losses Can Hallucinate Features in Medical Image Translation

Cohen, Joseph; Luck, Margaux; Honari, Sina

doi:10.1007/978-3-030-00928-1_60

Cited by 250 publications

(180 citation statements)

References 10 publications

Supporting

Mentioning

177

Contrasting

Order By: Relevance

“…In fact, cycle-GANbased methods have been shown to be limited in their ability to transform images even from one MR to a different MR sequence. 7 This approach overcomes the limitations of the prior approaches by incorporating structure-specific losses that constrain the cross-modality transformations to preserve atypical structures including tumors. We note however that all of these methods produce pseudo MR representations that do not create synthetic MR images modeling the same tissue specific intensities as the original MRI.…”

Section: Discussionmentioning

confidence: 99%

“…Additional testing was performed on a total of 39 MRI scans obtained from 22 patients who were not used in training or validation. Three of those patients contained longitudinal MRI scans (7,7,6). Table II contains details of the numbers of analyzed scans, patients, and ROIs (256 9 256) used in training and testing.…”

Section: E Implementation and Trainingmentioning

confidence: 99%

“…The pseudo MR images resemble real MR by mimicking the statistical intensity variations. To overcome the issues of the existing methods, which cannot accurately model the anatomical characteristics of atypical structures, including tumors, a tumor‐attention loss that regularizes the GAN model and produces pseudo MR with well‐preserved tumor structures is introduced. Figure shows example pseudo MR images generated from a CT image using the most recent cross‐modality adaptation method called the UNsupervised Image‐to‐Image Translation (UNIT), the CycleGAN, and the proposed approach.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cross‐modality (CT‐MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets

Jiang

Tyagi

et al. 2019

Medical Physics

View full text Add to dashboard Cite

Purpose Accurate tumor segmentation is a requirement for magnetic resonance (MR)‐based radiotherapy. Lack of large expert annotated MR datasets makes training deep learning models difficult. Therefore, a cross‐modality (MR‐CT) deep learning segmentation approach that augments training data using pseudo MR images produced by transforming expert‐segmented CT images was developed. Methods Eighty‐one T2‐weighted MRI scans from 28 patients with non‐small cell lung cancers (nine with pretreatment and weekly MRI and the remainder with pre‐treatment MRI scans) were analyzed. Cross‐modality model encoding the transformation of CT to pseudo MR images resembling T2w MRI was learned as a generative adversarial deep learning network. This model was used to translate 377 expert segmented non‐small cell lung cancer CT scans from the Cancer Imaging Archive into pseudo MRI that served as additional training set. This method was benchmarked against shallow learning using random forest, standard data augmentation, and three state‐of‐the art adversarial learning‐based cross‐modality data (pseudo MR) augmentation methods. Segmentation accuracy was computed using Dice similarity coefficient (DSC), Hausdorff distance metrics, and volume ratio. Results The proposed approach produced the lowest statistical variability in the intensity distribution between pseudo and T2w MR images measured as Kullback–Leibler divergence of 0.069. This method produced the highest segmentation accuracy with a DSC of (0.75 ± 0.12) and the lowest Hausdorff distance of (9.36 mm ± 6.00 mm) on the test dataset using a U‐Net structure. This approach produced highly similar estimations of tumor growth as an expert (P = 0.37). Conclusions A novel deep learning MR segmentation was developed that overcomes the limitation of learning robust models from small datasets by leveraging learned cross‐modality information using a model that explicitly incorporates knowledge of tumors in modality translation to augment segmentation training. The results show the feasibility of the approach and the corresponding improvement over the state‐of‐the‐art methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: E Implementation and Trainingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Cross‐modality (CT‐MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets

Jiang

Tyagi

et al. 2019

Medical Physics

View full text Add to dashboard Cite

show abstract

“…One difficulty with this approach is to ensure that the learned transformations sufficiently translate style while (1) remaining realistic, and (2) preserving content (anatomy) such that they are label-preserving [37]. As noted by Cohen et al [38], this is particularly problematic when using approaches based on adversarial losses (e.g. CycleGAN [34]) which aim to match the translation output with the distribution of the target domain, commonly introducing anatomical artifacts into the translated output, and introducing inconsistencies between the input labels and the transformed outputs.…”

Section: Discussionmentioning

confidence: 99%

Unsupervised domain adaptation for the automated segmentation of neuroanatomy in MRI: a deep learning approach

Novosad

Fonov

Collins

2019

Preprint

View full text Add to dashboard Cite

Neuroanatomical segmentation in T1-weighted magnetic resonance imaging of the brain is a prerequisite for quantitative morphological measurements, as well as an essential element in general preprocessing pipelines. While recent fully automated segmentation methods based on convolutional neural networks have shown great potential, these methods nonetheless suffer from severe performance degradation when there are mismatches between training (source) and testing (target) domains (e.g. due to different scanner acquisition protocols or due to anatomical differences in the respective populations under study). This work introduces a new method for unsupervised domain adaptation which improves performance in challenging cross-domain applications without requiring any additional annotations on the target domain. Using a previously validated state-of-the-art segmentation method based on a context-augmented convolutional neural network, we first demonstrate that networks with better domain generalizability can be trained using extensive data augmentation with label-preserving transformations which mimic differences between domains. Second, we incorporate unlabelled target domain samples into training using a self-ensembling approach, demonstrating further performance gains, and further diminishing the performance gap in comparison to fully-supervised training on the target domain.

show abstract

“…However, if proper care is not taken to represent truthfully the distribution of the data (e.g., not including enough tumor samples in a harmonization task between datasets with pathological data), this can lead to severe issues. Cohen et al (2018) recently showed that in such a case, GAN based methods could try to remove the pathology in the data to match the distribution of healthy subjects that the method previously learned, precluding potential applications to new datasets or pathological cases not represented "well enough" in the training set. The same concept would likely apply to systematic artifacts; if every dataset from a target scanner is corrupted by, e.g., a table vibration artifact, it may very well be possible that a harmonization algorithm will try to imprint this artifact to the source datasets to match the target datasets.…”

Section: Limitationsmentioning

confidence: 99%

Harmonization of diffusion MRI datasets with adaptive dictionary learning

St-Jean

Viergever

Leemans

2019

Preprint

View full text Add to dashboard Cite

Diffusion magnetic resonance imaging is a noninvasive imaging technique which can indirectly infer the microstructure of tissues and provide metrics which are subject to normal variability across subjects. Potentially abnormal values or features may yield essential information to support analysis of controls and patients cohorts, but subtle confounds affecting diffusion MRI, such as those due to difference in scanning protocols or hardware, can lead to systematic errors which could be mistaken for purely biologically driven variations amongst subjects. In this work, we propose a new harmonization algorithm based on adaptive dictionary learning to mitigate the unwanted variability caused by different scanner hardware while preserving the natural biological variability present in the data. Overcomplete dictionaries, which are learned automatically from the data and do not require paired samples, are then used to reconstruct the data from a different scanner, removing variability present in the source scanner in the process. We use the publicly available database from an international challenge to evaluate the method, which was acquired on three different scanners and with two different protocols, and propose a new mapping towards a scanner-agnostic space. Results show that the effect size of the four studied diffusion metrics is preserved while removing variability attributable to the scanner. Experiments with alterations using a free water compartment, which is not simulated in the training data, shows that the effect size induced by the alterations is also preserved after harmonization. The algorithm is freely available and could help multicenter studies in pooling their data, while removing scanner specific confounds, and increase statistical power in the process.

show abstract

Distribution Matching Losses Can Hallucinate Features in Medical Image Translation

Cited by 250 publications

References 10 publications

Cross‐modality (CT‐MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets

Cross‐modality (CT‐MRI) prior augmented deep learning for robust lung tumor segmentation from small MR datasets

Unsupervised domain adaptation for the automated segmentation of neuroanatomy in MRI: a deep learning approach

Harmonization of diffusion MRI datasets with adaptive dictionary learning

Contact Info

Product

Resources

About