CyTran: Cycle-Consistent Transformers for Non-Contrast to Contrast CT Translation

Ristea, Nicolae-Cătălin; Miron, Andreea-Iuliana; Savencu, Olivian; Verga, Nicolae; Khan, Fahad Shahbaz; Ionescu, Radu Tudor

doi:10.48550/arxiv.2110.06400

Cited by 6 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The adversarial training forces the generator model to predict more realistic segmentation out- comes. In addition to vanilla-GAN, we have also utilized the CycleGAN [17], [13] approach to investigate the effect of cycle consistency constraint on the segmentation task. Figure 1 demonstrates the general overview of the proposed method.…”

Section: Proposed Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Transformer based Generative Adversarial Network for Liver Segmentation

Demir¹,

Zhang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automated liver segmentation from radiology scans (CT, MRI) can improve surgery and therapy planning and follow-up assessment in addition to conventional use for diagnosis and prognosis. Although convolutional neural networks (CNNs) have became the standard image segmentation tasks, more recently this has started to change towards Transformers based architectures because Transformers are taking advantage of capturing long range dependence modeling capability in signals, so called attention mechanism. In this study, we propose a new segmentation approach using a hybrid approach combining the Transformer(s) with the Generative Adversarial Network (GAN) approach. The premise behind this choice is that the self-attention mechanism of the Transformers allows the network to aggregate the high dimensional feature and provide global information modeling. This mechanism provides better segmentation performance compared with traditional methods. Furthermore, we encode this generator into the GAN based architecture so that the discriminator network in the GAN can classify the credibility of the generated segmentation masks compared with the real masks coming from human (expert) annotations. This allows us to extract the high dimensional topology information in the mask for biomedical image segmentation and provide more reliable segmentation results. Our model achieved a high dice coefficient of 0.9433, recall of 0.9515, and precision of 0.9376 and outperformed other Transformer based approaches. The implementation details of the proposed architecture can be found at https://github.com/UgurDemir/tranformer_liver_segmentation.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

“…To embed transformers within the CycleGAN, we utilized the encoder-decoder style convolutional transformer model [13]. The premise behind this idea was that the encoder module takes the input image and decreases the spatial dimensions while extracting features with convolution layers.…”

Section: Transformer Based Cycleganmentioning

confidence: 99%

Transformer based Generative Adversarial Network for Liver Segmentation

Demir¹,

Zhang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In our work, we aim to benefit from the modeling power of transformers while being able to process reasonable downsampled ocean SAR images, we adopt a generative convolutional transformer with a manageable number of parameters called CyTran [31]. We used it in an unsupervised set-up, showing that, using SD as a preprocessing stage we improve the SAR image descriptors, leading to an important precision boost for image retrieval.…”

Section: Transformer Modelsmentioning

confidence: 99%

“…The downsampling block is followed by the convolutional transformer block, which provides an output tensor of the same size as the input tensor. The convolutional transformer block is inspired by the block proposed in [31]. More precisely, the input tensor is interpreted as a set of overlapping visual tokens.…”

Section: Unsupervised Neural Networkmentioning

confidence: 99%

Guided Deep Learning by Subaperture Decomposition: Ocean Patterns from SAR Imagery

Ristea

Anghel

Datcu

et al. 2022

IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium

Self Cite

View full text Add to dashboard Cite

Spaceborne synthetic aperture radar (SAR) can provide accurate images of the ocean surface roughness dayor-night in nearly all weather conditions, being an unique asset for many geophysical applications. Considering the huge amount of data daily acquired by satellites, automated techniques for physical features extraction are needed. Even if supervised deep learning methods attain state-of-the-art results, they require great amount of labeled data, which are difficult and excessively expensive to acquire for ocean SAR imagery. To this end, we use the subaperture decomposition (SD) algorithm to enhance the unsupervised learning retrieval on the ocean surface, empowering ocean researchers to search into large ocean databases. We empirically prove that SD improve the retrieval precision with over 20% for an unsupervised transformer auto-encoder network. Moreover, we show that SD brings important performance boost when Doppler centroid images are used as input data, leading the way to new unsupervised physics guided retrieval algorithms.

show abstract

“…Moreover, using transformers has been shown to be more promising in computer vision (Dosovitskiy et al, 2020; for utilizing long-range dependencies than other, traditional CNN-based methods. In parallel, transformers with powerful global relation modeling abilities have become the standard starting point for training on a wide range of downstream medical imaging analysis tasks, such as image segmentation Cao et al, 2021;Wang et al, 2021b;Valanarasu et al, 2021;Xie et al, 2021b), image synthesis (Kong et al, 2021;Ristea et al, 2021;Dalmaz et al, 2021), and image enhancement (Korkmaz et al, 2021;Luthra et al, 2021;Wang et al, 2021a).…”

Section: Introductionmentioning

confidence: 99%

Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

You,

Zhao,

Liu

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CA-GANformer, a novel type of generative adversarial transformers, for medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CA-GANformer dramatically outperforms previous state-of-the-art transformerbased approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model's inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CA-GANformer a strong starting point for downstream medical im-

show abstract

CyTran: Cycle-Consistent Transformers for Non-Contrast to Contrast CT Translation

Cited by 6 publications

References 0 publications

Transformer based Generative Adversarial Network for Liver Segmentation

Transformer based Generative Adversarial Network for Liver Segmentation

Guided Deep Learning by Subaperture Decomposition: Ocean Patterns from SAR Imagery

Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

Contact Info

Product

Resources

About