PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer

Zhang, Xuzhe; He, Xinzi; Guo, Jia; Ettehadi, Nabil; Aw, Natalie; Semanek, David; Posner, Jonathan; Laine, Andrew F.; Wang, Yun

doi:10.48550/arxiv.2105.13993

Cited by 7 publications

(19 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Paired data is difficult to obtain due to annotation cost and time constraints, thereby generally hindering the applicability of these models in medical imaging applications. Zhang et al [315] focus on synthesizing infant brain structural MRIs (T1w and T2w scans) using both transformer and performer (simplified selfattention) layers [88]. Specifically, they design a novel multiresolution pyramid-like U-Net framework, PTNet, utilizing performer encoder, performer decoder, and transformer bottleneck to synthesize high-quality infant MRI.…”

Section: Supervised Methodsmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Supervised Methodsmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The pyramid layer might provide such an improvement as long as it runs at a reasonable resolution. 3) 2D and 3D Comparison: Lastly, we conducted an ablation study between the proposed PTNet3D and our previous 2D variant [60]. The results were listed below.…”

Section: Ablation Studiesmentioning

confidence: 99%

“…Random motion artifacts were introduced using approaches developed in previous studies [60], [61]. Details and examples of motion artifact injection were provided in Append i x G. These datasets which contain different ratios of corrupted scans were fed into the pre-trained segmentation model described in Section IV.C.…”

Section: F Application-prevent Data Exclusion In Downstream Tasks By ...mentioning

confidence: 99%

PTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers

Zhang

Guo

et al. 2022

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

An increased interest in longitudinal neurodevelopment during the first few years after birth has emerged in recent years. Noninvasive magnetic resonance imaging (MRI) can provide crucial information about the development of brain structures in the early months of life. Despite the success of MRI collections and analysis for adults, it remains a challenge for researchers to collect high-quality multimodal MRIs from developing infant brains because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still during scanning. In addition, there are limited analytic approaches available. These challenges often lead to a significant reduction of usable MRI scans and pose a problem for modeling neurodevelopmental trajectories. Researchers have explored solving this problem by synthesizing realistic MRIs to replace corrupted ones. Among synthesis methods, the convolutional neural network-based (CNN-based) generative adversarial networks (GANs) have demonstrated promising performance. In this study, we introduced a novel 3D MRI synthesis framework -pyramid transformer network (PTNet3D) -which relies on attention mechanisms through transformer and performer layers. We conducted extensive experiments on high-resolution Developing Human Connectome Project (dHCP) and longitudinal Baby Connectome Project (BCP) datasets. Compared with CNN-based GANs, PTNet3D consistently shows superior synthesis accuracy and superior generalization on two independent, large-scale infant brain MRI datasets. Notably, we demonstrate that PTNet3D synthesized more realistic scans than CNN-based models when the input is from multi-age subjects. Potential applications of PTNet3D include synthesizing corrupted or Manuscript

show abstract

“…In the domain of MRI, Zhang et al [38] developed PTNet for synthesizing infant MRI images. Feng et al proposed T 2 Net [39] for joint MRI reconstruction and super-resolution.…”

Section: B Transformermentioning

confidence: 99%

One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation

Liu¹,

Pasumarthi²,

Duffy³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each contrast may vary amongst patients in reality. This poses challenges to both radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural network (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as requirement for fixed numbers of input and output channels, inability to capture long-range dependencies, and lack of interpretability. In this paper, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. MMT consists of a multiscale Transformer encoder that builds hierarchical representations of inputs combined with a multi-scale Transformer decoder that generates the outputs in a coarse-tofine fashion. Thanks to the proposed multi-contrast Swin Transformer blocks, it can efficiently capture intra-and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable. It allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of Transformer blocks in the decoder. Extensive experiments on two large-scale multi-contrast MRI datasets demonstrate that MMT outperforms the state-of-the-art methods quantitatively and qualitatively.

show abstract

PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer

Cited by 7 publications

References 30 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

PTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers

One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation

Contact Info

Product

Resources

About