GANBERT: Generative Adversarial Networks with Bidirectional Encoder Representations from Transformers for MRI to PET synthesis

Shin, Hoo-Chang; Ihsani, Alvin; Mandava, Swetha; Sreenivas, Sharath Turuvekere; Forster, Christopher; Cha, Jiook; Initiative, Alzheimer's Disease Neuroimaging

doi:10.48550/arxiv.2008.04393

Cited by 8 publications

(17 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Transformer models have also been verified to have powerful learning capabilities in various medical image enhancement tasks, including image super-resolution [114], denoising [115], reconstruction [116], synthesis [117], and registration [118]. Feng et al [114] introduced an end-to-end task transformer network termed T2Net, allowing feature representations to be shared and transferred between MRI reconstruction and super-resolution tasks.…”

Section: Transformer In Medical Image Enhancement Taskmentioning

confidence: 99%

Deep Learning Attention Mechanism in Medical Image Analysis: Basics and Beyonds

Li²,

Yan³

et al. 2023

IJNDI

View full text Add to dashboard Cite

Survey/review study Deep Learning Attention Mechanism in Medical Image Analysis: Basics and Beyonds Xiang Li 1, Minglei Li 1, Pengfei Yan 1, Guanyi Li 1, Yuchen Jiang 1, Hao Luo 1,*, and Shen Yin 2 1 Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China 2 Department of Mechanical and Industrial Engineering, Faculty of Engineering, Norwegian University of Science and Technology, Trondheim 7034, Norway * Correspondence: hao.luo@hit.edu.cn Received: 16 October 2022 Accepted: 25 November 2022 Published: Abstract: With the improvement of hardware computing power and the development of deep learning algorithms, a revolution of "artificial intelligence (AI) + medical image" is taking place. Benefiting from diversified modern medical measurement equipment, a large number of medical images will be produced in the clinical process. These images improve the diagnostic accuracy of doctors, but also increase the labor burden of doctors. Deep learning technology is expected to realize an auxiliary diagnosis and improve diagnostic efficiency. At present, the method of deep learning technology combined with attention mechanism is a research hotspot and has achieved state-of-the-art results in many medical image tasks. This paper reviews the deep learning attention methods in medical image analysis. A comprehensive literature survey is first conducted to analyze the keywords and literature. Then, we introduce the development and technical characteristics of the attention mechanism. For its application in medical image analysis, we summarize the related methods in medical image classification, segmentation, detection, and enhancement. The remaining challenges, potential solutions, and future research directions are also discussed.

show abstract

Section: Transformer In Medical Image Enhancement Taskmentioning

confidence: 99%

Deep Learning Attention Mechanism in Medical Image Analysis: Basics and Beyonds

Li²,

Yan³

et al. 2023

IJNDI

View full text Add to dashboard Cite

show abstract

“…Transformers in data augmentation has been explored in Zhang et al [16] which utilises self attention from transformers to explore long range dependencies in internal representation of images. Another network focusing on synthesizing images is the GANBERT or GAN with a bidirectional encoder representation from transformer [11] which uses self attention to generate difficult medical images like that from MRI and PET scans.…”

Section: Related Workmentioning

confidence: 99%

Transformer Networks for Data Augmentation of Human Physical Activity Recognition

Ramachandra,

Hoelzemann,

Van Laerhoven

2021

Preprint

View full text Add to dashboard Cite

Data augmentation is a widely used technique in classification to increase data used in training. It improves generalization and reduces amount of annotated human activity data needed for training which reduces labour and time needed with the dataset. Sensor time-series data, unlike images, cannot be augmented by computationally simple transformation algorithms. State of the art models like Recurrent Generative Adversarial Networks (RGAN) are used to generate realistic synthetic data. In this paper, transformer based generative adversarial networks which have global attention on data, are compared on PAMAP2 and Real World Human Activity Recognition data sets with RGAN. The newer approach provides improvements in time and savings in computational resources needed for data augmentation than previous approach. CCS CONCEPTS• Computing methodologies → Neural networks; Machine learning algorithms; • Human-centered computing → Ubiquitous computing.

show abstract

“…The second branch comprises a multi-scale 3D convolutional network that integrates multi-parametric MRI images to distinguish between healthy controls and people with cerebrovascular diseases. Results show that the proposed In the past three years, several deep convolutional neural networks have been introduced to predict PET CBF maps from structural and perfusion MRI images [11], [19], [20]. In [11], Guo et al adopted a deep CNN (dCNN) to generate synthetic 15 O-water PET CBF images from multi-parametric MRI inputs including ASL.…”

Section: Introductionmentioning

confidence: 98%

“…The dCNN notably improved CBF image quality, when compared to ASL, achieving an average structural similarity index (SSIM) of 0.85. In [19], Shin et al studied the possibility of synthesizing different brain PET tracers (specifically AV45, AV1451, fluorodeoxyglucose) solely from T1-weighted MRI images using generative adversarial networks (GANs). This method achieved limited PET prediction results, with an average SSIM of 0.26-0.38.…”

Section: Introductionmentioning

confidence: 99%

Multi-task Deep Learning for Cerebrovascular Disease Classification and MRI-to-PET Translation

Hussein¹,

Zhao²,

Shin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of cerebrovascular diseases such as Moyamoya, carotid stenosis, aneurysms, and stroke. Positron emission tomography (PET) is currently regarded as the gold standard for the measurement of CBF in the human brain. PET imaging, however, is not widely available because of its prohibitive costs, use of ionizing radiation, and logistical challenges, which require a co-localized cyclotron to deliver the 2 min half-life 15 O radioisotope. Magnetic resonance imaging (MRI), in contrast, is more readily available and does not involve ionizing radiation. In this study, we propose a multitask learning framework for brain MRI-to-PET translation and disease diagnosis. The proposed framework comprises two prime networks: (1) an attention-based 3D encoder-decoder convolutional neural network (CNN) that synthesizes high-quality PET CBF maps from multi-contrast MRI images, and (2) a multi-scale 3D CNN that identifies the brain disease corresponding to the input MRI images. Our multi-task framework yields promising results on the task of MRI-to-PET translation, achieving an average structural similarity index (SSIM) of 0.94 and peak signal-to-noise ratio (PSNR) of 38dB on a cohort of 120 subjects. In addition, we show that integrating multiple MRI modalities can improve the clinical diagnosis of brain diseases.

show abstract

GANBERT: Generative Adversarial Networks with Bidirectional Encoder Representations from Transformers for MRI to PET synthesis

Cited by 8 publications

References 15 publications

Deep Learning Attention Mechanism in Medical Image Analysis: Basics and Beyonds

Deep Learning Attention Mechanism in Medical Image Analysis: Basics and Beyonds

Transformer Networks for Data Augmentation of Human Physical Activity Recognition

Multi-task Deep Learning for Cerebrovascular Disease Classification and MRI-to-PET Translation

Contact Info

Product

Resources

About