Joint Registration and Segmentation via Multi-Task Learning for Adaptive Radiotherapy of Prostate Cancer

Abstract

Medical image registration and segmentation are two of the most frequent tasks in medical image analysis. As these tasks are complementary and correlated, it would be beneficial to apply them simultaneously in a joint manner. In this paper, we formulate registration and segmentation as a joint problem via a Multi-Task Learning (MTL) setting, allowing these tasks to leverage their strengths and mitigate their weaknesses through the sharing of beneficial information. We propose to merge these tasks not only on the loss level, but on the architectural level as well. We studied this approach in the context of adaptive image-guided radiotherapy for prostate cancer, where planning and follow-up CT images as well as their corresponding contours are available for training. At testing time the contours of the follow-up scans are not available, which is a common scenario in adaptive radiotherapy. The study involves two datasets from different manufacturers and institutes. The first dataset was divided into training (12 patients) and validation (6 patients), and was used to optimize and validate the methodology, while the second dataset (14 patients) was used as an independent test set. We carried out an extensive quantitative comparison between the quality of the automatically generated contours from different network architectures as well as loss weighting methods. Moreover, we evaluated the quality of the generated deformation vector field (DVF). We show that MTL algorithms outperform their Single-Task Learning (STL) counterparts and achieve better generalization on the independent test set. The best algorithm achieved a mean surface distance of 1.06 ± 0.3 mm, 1.27 ± 0.4 mm, 0.91 ± 0.4 mm, and 1.76 ± 0.8 mm on the validation set for the prostate, seminal vesicles, bladder, and rectum, respectively. The high accuracy of the proposed method combined with the fast inference speed, makes it a promising method for automatic re-contouring of follow-up scans for adaptive radiotherapy, potentially reducing treatment related complications and therefore improving patients quality-of-life after treatment.

Deep‐learning‐based joint rigid and deformable contour propagation for magnetic resonance imaging‐guided prostate radiotherapy

“…14 Combining the advantages of registration and segmentation to obtain fused models could also be powerful. 41 We identified the conditions under which the framework achieves "acceptable" contour accuracy-defined as a minimum prostate Dice of 0.67 (25th percentile of the iterative rigid registration) for the rigid registration and 0.86 (considered acceptable by a radiation oncologist 38 ) for the deformable registration. The framework comprising LapIRN achieved acceptable contour accuracies for translations up to 52 mm, rotations up to 15 • , nonlinear deformations up to 40 mm, and bias fields to some extent.…”

“…For example, weak supervision with the contours could achieve this improvement 14 . Combining the advantages of registration and segmentation to obtain fused models could also be powerful 41 …”

Contour‐guided deep learning based deformable image registration for dose monitoring during CBCT‐guided radiotherapy of prostate cancer

“…Moreover, while the proposed methods exploit delineations as inputs, this approach may be improved using multi-tasking learning strategies to couple the tasks of automatic segmentation and contour propagation via DVF estimation, improving both results using prior patient specific information. 26,27 To our knowledge, no other study investigated the usage of DL-based DIR to perform pelvic dose accumulation during prostate radiotherapy. However, previous studies investigated the variability in estimating the delivered dose.…”

“…Future DL‐based strategies could integrate a determinant of Jacobian penalty to improve DVF inversibility. Moreover, while the proposed methods exploit delineations as inputs, this approach may be improved using multi‐tasking learning strategies to couple the tasks of automatic segmentation and contour propagation via DVF estimation, improving both results using prior patient specific information 26,27 …”

macJNet: weakly-supervised multimodal image deformable registration using joint learning framework and multi-sampling cascaded MIND

“…The main aim of this work is to propose a general framework for weakly-supervised registration, any task-specific registration or segmentation networks could be used as sub-networks in this framework. Some other works [ 27 , 38 ] joint the registration and segmentation through multi-task learning. Multi-task learning methods joint the two tasks using hard or soft parameter sharing, which needs to change the architecture of existing networks.…”