PTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers

Abstract

An increased interest in longitudinal neurodevelopment during the first few years after birth has emerged in recent years. Noninvasive magnetic resonance imaging (MRI) can provide crucial information about the development of brain structures in the early months of life. Despite the success of MRI collections and analysis for adults, it remains a challenge for researchers to collect high-quality multimodal MRIs from developing infant brains because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still during scanning. In addition, there are limited analytic approaches available. These challenges often lead to a significant reduction of usable MRI scans and pose a problem for modeling neurodevelopmental trajectories. Researchers have explored solving this problem by synthesizing realistic MRIs to replace corrupted ones. Among synthesis methods, the convolutional neural network-based (CNN-based) generative adversarial networks (GANs) have demonstrated promising performance. In this study, we introduced a novel 3D MRI synthesis framework -pyramid transformer network (PTNet3D) -which relies on attention mechanisms through transformer and performer layers. We conducted extensive experiments on high-resolution Developing Human Connectome Project (dHCP) and longitudinal Baby Connectome Project (BCP) datasets. Compared with CNN-based GANs, PTNet3D consistently shows superior synthesis accuracy and superior generalization on two independent, large-scale infant brain MRI datasets. Notably, we demonstrate that PTNet3D synthesized more realistic scans than CNN-based models when the input is from multi-age subjects. Potential applications of PTNet3D include synthesizing corrupted or Manuscript

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

“…introduced SuperFormer, which integrates 3D ViTs to effectively utilize 3D anatomical information. Similarly, Zhang et al [87] . suggested an SR synthesizer based on a pyramid transformer and implemented it in the synthesis of infant brain MRI.…”

SCANeXt: Enhancing 3D medical image segmentation with dual attention network and depth-wise convolution

“…In recent years, vision transformers (ViTs) [1] have gradually surpassed and replaced Convolution Neural Network (CNN) and found wide applications in various downstream tasks of medical imaging, including segmentation [2] , [3] , [4] , [5] , classification [6] , [7] , [8] , [9] , restoration [10] , [11] , [12] , [13] , synthesis [14] , [15] , [16] , [17] , registration [18] , [19] , [20] , [21] , and object detection in medical images [22] , [23] . In particular, significant progress has been observed in 3D medical image segmentation with the adoption of Vision Transformers (ViTs) [24] , [25] , [26] , [27] , [28] .…”

Deformable 3D medical image registration with convolutional neural network and transformer

“…Recently, transformers have become the focus of research in medical image analysis, and achieved better performance than CNN. More and more variants of the transformer are used as backbone networks in medical image segmentation [2,3], gene identification [4], Anomaly detection [5], COVID-19 detection [6], and image synthesis [7].…”