Synthetic cranial MRI from 3D optical surface scans using deep learning for radiation therapy treatment planning

Abstract

Background Optical scanning technologies are increasingly being utilised to supplement treatment workflows in radiation oncology, such as surface-guided radiotherapy or 3D printing custom bolus. One limitation of optical scanning devices is the absence of internal anatomical information of the patient being scanned. As a result, conventional radiation therapy treatment planning using this imaging modality is not feasible. Deep learning is useful for automating various manual tasks in radiation oncology, most notably, organ segmentation and treatment planning. Deep learning models have also been used to transform MRI datasets into synthetic CT datasets, facilitating the development of MRI-only radiation therapy planning. Aims To train a pix2pix generative adversarial network to transform 3D optical scan data into estimated MRI datasets for a given patient to provide additional anatomical data for a select few radiation therapy treatment sites. The proposed network may provide useful anatomical information for treatment planning of surface mould brachytherapy, total body irradiation, and total skin electron therapy, for example, without delivering any imaging dose. Methods A 2D pix2pix GAN was trained on 15,000 axial MRI slices of healthy adult brains paired with corresponding external mask slices. The model was validated on a further 5000 previously unseen external mask slices. The predictions were compared with the “ground-truth” MRI slices using the multi-scale structural similarity index (MSSI) metric. A certified neuro-radiologist was subsequently consulted to provide an independent review of the model’s performance in terms of anatomical accuracy and consistency. The network was then applied to a 3D photogrammetry scan of a test subject to demonstrate the feasibility of this novel technique. Results The trained pix2pix network predicted MRI slices with a mean MSSI of 0.831 ± 0.057 for the 5000 validation images indicating that it is possible to estimate a significant proportion of a patient’s gross cranial anatomy from a patient’s exterior contour. When independently reviewed by a certified neuro-radiologist, the model’s performance was described as “quite amazing, but there are limitations in the regions where there is wide variation within the normal population.” When the trained network was applied to a 3D model of a human subject acquired using optical photogrammetry, the network could estimate the corresponding MRI volume for that subject with good qualitative accuracy. However, a ground-truth MRI baseline was not available for quantitative comparison. Conclusions A deep learning model was developed, to transform 3D optical scan data of a patient into an estimated MRI volume, potentially increasing the usefulness of optical scanning in radiation therapy planning. This work has demonstrated that much of the human cranial anatomy can be predicted from the external shape of the head and may provide an additional source of valuable imaging data. Further research is required to investigate the feasibility of this approach for use in a clinical setting and further improve the model’s accuracy.

BrainPixGAN: Generating Intraoperative MR with Mask-Based Generative Networks

“…As a result of their study, they reached a SSIM value of 0.3623 when comparing the synthesized MR image to the actual MR image [14]. Douglass et al (2023) utilized deep learning techniques to overcome limitations of optical scanning in radiation oncology treatment planning. Optical scanners depict external anatomy but lack internal details, prompting their study to use Pix2Pix GAN models with optical brain scans for internal structure prediction.…”

“…Results showed successful cranial anatomy prediction yet limitations in some areas. Additionally, predictions on 3D models from optical photogrammetry were qualitatively accurate but lacked MRI data for verification [15].…”

An evaluation of consumer smartphones for generating bolus and surface mould applicators for radiation oncology

“…As a result, a plethora of radiotherapy tools ranging from bolus, 1–3 brachytherapy applicators, 4,5 and shielding materials 6 can be designed with relative ease. Their use is not just limited to 3D printable devices but have been extended to generating pseudo‐ Computed Tomography (CT) water equivalent datasets for use in certain treatment planning techniques 1,7,8 and patient positioning 9,10 …”

“…As a result, a plethora of radiotherapy tools ranging from bolus, [1][2][3] brachytherapy applicators, 4,5 and shielding materials 6 can be designed with relative ease. Their use is not just limited to 3D printable devices but have been extended to generating pseudo-Computed Tomography (CT) water equivalent datasets for use in certain treatment planning techniques 1,7,8 and patient positioning. 9,10 These imaging technologies exhibit several advantages over the more commonly used method of CT scanning for generating 3D models, which demonstrates their feasibility for use in a radiation oncology department.…”