MR to ultrasound image registration with segmentation‐based learning for HDR prostate brachytherapy

Abstract

Purpose Propagation of contours from high‐quality magnetic resonance (MR) images to treatment planning ultrasound (US) images with severe needle artifacts is a challenging task, which can greatly aid the organ contouring in high dose rate (HDR) prostate brachytherapy. In this study, a deep learning approach was developed to automatize this registration procedure for HDR brachytherapy practice. Methods Because of the lack of training labels and difficulty of accurate registration from inferior image quality, a new segmentation‐based registration framework was proposed for this multi‐modality image registration problem. The framework consisted of two segmentation networks and a deformable registration network, based on the weakly ‐supervised registration strategy. Specifically, two 3D V‐Nets were trained for the prostate segmentation on the MR and US images separately, to generate the weak supervision labels for the registration network training. Besides the image pair, the corresponding prostate probability maps from the segmentation were further fed to the registration network to predict the deformation matrix, and an augmentation method was designed to randomly scale the input and label probability maps during the registration network training. The overlap between the deformed and fixed prostate contours was analyzed to evaluate the registration accuracy. Three datasets were collected from our institution for the MR and US image segmentation networks, and the registration network learning, which contained 121, 104, and 63 patient cases, respectively. Results The mean Dice similarity coefficient (DSC) results of the two prostate segmentation networks are 0.86 ± 0.05 and 0.90 ± 0.03, for MR images and the US images after the needle insertion, respectively. The mean DSC, center‐of‐mass (COM) distance, Hausdorff distance (HD), and averaged symmetric surface distance (ASSD) results for the registration of manual prostate contours were 0.87 ± 0.05, 1.70 ± 0.89 mm, 7.21 ± 2.07 mm, 1.61 ± 0.64 mm, respectively. By providing the prostate probability map from the segmentation to the registration network, as well as applying the random map augmentation method, the evaluation results of the four metrics were all improved, such as an increase in DSC from 0.83 ± 0.08 to 0.86 ± 0.06 and from 0.86 ± 0.06 to 0.87 ± 0.05, respectively. Conclusions A novel segmentation‐based registration framework was proposed to automatically register prostate MR images to the treatment planning US images with metal artifacts, which not only largely saved the labor work on the data preparation, but also improved the registration accuracy. The evaluation results showed the potential of this approach in HDR prostate brachytherapy practice.

A segmentation-informed deep learning framework to register dynamic two-dimensional magnetic resonance images of the vocal tract during speech

“…First, the framework includes a method to segment the images. Second, segmentations as well as images are used as inputs to the registration CNN, in the same manner as the framework of Chen et al [43] . Incorporation of a segmentation method in the framework enables its use when segmentations of the images are not already available.…”

“…First, it includes a method to segment the images. Second, segmentations as well as images are used as inputs to the registration CNN, in the same manner as the framework of Chen et al [43] . Fig.…”

“…Registration and segmentation can be related tasks, and there is increasing evidence that including segmentation information during the training of a registration CNN results in more accurate motion estimates [37] , [42] , [43] , [44] , [45] , [46] , [47] , [48] , [49] , [50] , [51] . Inclusion of such information is typically achieved by including region-overlap-based terms such as the Dice coefficient (DSC) in the CNN loss function.…”

“…In this approach, the images are modified before being used as inputs to the registration CNNs either by multiplying them by binary segmentations [49] , [50] or by using a fully convolutional image transformer network whose loss function includes a region-overlap-based term [51] . The second approach is to use segmentations as well as images as inputs to the registration CNN [43] . The rationale for inputting segmentations, even if these are estimates rather than ground-truths, is that they provide information about the positions of anatomical features in the images and would therefore help the CNN to estimate more accurate displacement fields.…”

Machine Learning in Prostate MRI for Prostate Cancer: Current Status and Future Opportunities

“…The method produced a mean TRE of 2.53 mm. Similarly, Chen et al [ 74 ] in 2021 also employed a segmentation model before the registration of MR images to the treatment planning US images after the needle insertion for HDR brachytherapy.…”