Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation

Tomar, Devavrat; Bozorgtabar, Behzad; Lortkipanidze, Manana; Vray, Guillaume; Rad, Mohammad Saeed; Thiran, Jean‐Philippe

doi:10.1109/wacv51458.2022.00180

Cited by 23 publications

(19 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors of (Denck et al, 2021), for example, assert that contrast differentiation across MRI modalities would result in a style variation, therefor, they presented an image-to-image generative adversarial network capable of synthesizing MR pictures with configurable contrast via style transfer. The network presented by the authors of (Tomar et al, 2022) also employ a style encoder to group images with roughly similar styles together. According to (Ma et al, 2019), utilizing neural style transfer to overcome existing inconsistencies in brightness, contrast, and texture, all of which are considered style inconsistencies, the network's segmentation performance would enhance.…”

Section: Appendix a Clinical Effect Of Missing Modalitiesmentioning

confidence: 99%

SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities

Azad¹,

Khosravi²,

Merhof³

2022

Preprint

View full text Add to dashboard Cite

Gliomas are one of the most prevalent types of primary brain tumors, accounting for more than 30% of all cases and they develop from the glial stem or progenitor cells. In theory, the majority of brain tumors could well be identified exclusively by the use of Magnetic Resonance Imaging (MRI). Each MRI modality delivers distinct information on the soft tissue of the human brain and integrating all of them would provide comprehensive data for the accurate segmentation of the glioma, which is crucial for the patient's prognosis, diagnosis, and determining the best follow-up treatment. Unfortunately, MRI is prone to artifacts for a variety of reasons, which might result in missing one or more MRI modalities. Various strategies have been proposed over the years to synthesize the missing modality or compensate for the influence it has on automated segmentation models. However, these methods usually fail to model the underlying missing information. In this paper, we propose a style matching U-Net (SMU-Net) for brain tumour segmentation on MRI images. Our co-training approach utilizes a content and style-matching mechanism to distill the informative features from the full-modality network into a missing modality network. To do so, we encode both full-modality and missing-modality data into a latent space, then we decompose the representation space into a style and content representation. Our style matching module adaptively recalibrates the representation space by learning a matching function to transfer the informative and textural features from full-modality path into a missing-modality path. Moreover, by modelling the mutual information, our content module surpasses the less informative features and re-calibrates the representation space based on discriminative semantic features. The evaluation process on the BraTS 2018 dataset shows the significance of the proposed method on the missing modality scenario.

show abstract

Section: Appendix a Clinical Effect Of Missing Modalitiesmentioning

confidence: 99%

SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities

Azad¹,

Khosravi²,

Merhof³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…ART-type Applications. In recent decades, a category of typical learning tasks towards sophisticated applications have addressed considered learning tasks with related auxiliary learning devices, named auxiliary with related tasks, such as medical image analysis (i.e., medical image registration and segmentation [26], [27], [28], [29] and lowlight image enhancement [30], [31]) and hyper-parameter learning [5], [13], [14], [32], [33]. OL and CL can be regarded as objective learning task and auxiliary learning task, respectively.…”

Section: Review Of Related Work Within Lwclmentioning

confidence: 99%

“…and CPS [56]. d) joint registration and segmentation methods, SST [28], DeepAtlas [26], DataAug [29], and BRBS [27].…”

Section: Medical Image Analysismentioning

confidence: 99%

Special Session: Fault-Tolerant Deep Learning: A Hierarchical Perspective

Liu

Gao

Liu

et al. 2022

2022 IEEE 40th VLSI Test Symposium (VTS)

View full text Add to dashboard Cite

The complexity of learning problems, such as Generative Adversarial Network (GAN) and its variants, multi-task and meta-learning, hyper-parameter learning, and a variety of real-world vision applications, demands a deeper understanding of their underlying coupling mechanisms. Existing approaches often address these problems in isolation, lacking a unified perspective that can reveal commonalities and enable effective solutions. Therefore, in this work, we proposed a new framework, named Learning with Constraint Learning (LwCL), that can holistically examine challenges and provide a unified methodology to tackle all the above-mentioned complex learning and vision problems. Specifically, LwCL is designed as a general hierarchical optimization model that captures the essence of these diverse learning and vision problems. Furthermore, we develop a gradient-response based fast solution strategy to overcome optimization challenges of the LwCL framework. Our proposed framework efficiently addresses a wide range of applications in learning and vision, encompassing three categories and nine different problem types. Extensive experiments on synthetic tasks and real-world applications verify the effectiveness of our approach. The LwCL framework offers a comprehensive solution for tackling complex machine learning and computer vision problems, bridging the gap between theory and practice.

show abstract

“…Additionally, image generation can be constrained by the appearance of the anatomical structures and segmentation maps. Many approaches have been presented in the literature that generate image-mask pairs, for instance, implementing domain adaptation from CT to MR [17], generating synthetic samples to solve a segmentation task [18,19,20,21] or for one-shot segmentation [22,23,24].…”

Section: Introductionmentioning

confidence: 99%

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

Franco-Barranco¹,

Pastor-Tronch²,

Gonzalez-Marfil³

et al. 2022

Preprint

View full text Add to dashboard Cite

Background and Objective: Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species. Methods: We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-ofthe-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation Results: We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets. Conclusions: The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels,

show abstract

Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation

Cited by 23 publications

References 73 publications

SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities

SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities

Special Session: Fault-Tolerant Deep Learning: A Hierarchical Perspective

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

Contact Info

Product

Resources

About