Adversarial Deformation Regularization for Training Image Registration Neural Networks

Abstract

We describe an adversarial learning approach to constrain convolutional neural network training for image registration, replacing heuristic smoothness measures of displacement fields often used in these tasks. Using minimallyinvasive prostate cancer intervention as an example application, we demonstrate the feasibility of utilizing biomechanical simulations to regularize a weakly-supervised anatomical-label-driven registration network for aligning pre-procedural magnetic resonance (MR) and 3D intra-procedural transrectal ultrasound (TRUS) images. A discriminator network is optimized to distinguish the registration-predicted displacement fields from the motion data simulated by finite element analysis. During training, the registration network simultaneously aims to maximize similarity between anatomical labels that drives image alignment and to minimize an adversarial generator loss that measures divergence between the predicted-and simulated deformation. The end-to-end trained network enables efficient and fully-automated registration that only requires an MR and TRUS image pair as input, without anatomical labels or simulated data during inference. 108 pairs of labelled MR and TRUS images from 76 prostate cancer patients and 71,500 nonlinear finite-element simulations from 143 different patients were used for this study. We show that, with only gland segmentation as training labels, the proposed method can help predict physically plausible deformation without any other smoothness penalty. Based on cross-validation experiments using 834 pairs of independent validation landmarks, the proposed adversarial-regularized registration achieved a target registration error of 6.3 mm that is significantly lower than those from several other regularization methods.

Deep learning in medical image registration: a survey

“…8. In another work, Hu et al [42] simultaneously maximized label similarity and minimized an adversarial loss term to predict the deformation for MR-TRUS registration. This regularization term forces the predicted transformation to result in the generation of a realistic image.…”

“…However, the use of such regularization terms may limit the magnitude of the deformations that neural networks are able to predict. Therefore, Hu et al [42] explored the use of a GAN-like framework to produce realistic deformations. Constraining the deformation prediction using a discriminator results in superior performance relative to the use of L2 norm regularization in that work.…”

LungRegNet: An unsupervised deformable image registration method for 4D‐CT lung

“…Hu et al trained an adversarial network to tell whether a transformation is predicted by the network or generated by finite element method (FEM). 46 The purpose of the adversarial network was to introduce biomechanical constraint to MR and TRUS prostate registration. Another way of utilizing GAN in image registration is to cast multimodal to unimodal image registration via image synthesis.…”

“…The advantage is that we could eliminate the requirement of ground truth image alignment by discriminating on images instead of image pairs. For example, ground truth transformations were required in Yan et al's work, 44 prealigned MR-CT image pairs were required in Fan et al's work, 45 and FEM-generated transformations were required in Hu et al's work 46 . In terms of unsupervised learning, recently published Jiang et al's work 36 is most relevant to our work, however, they did not use adversarial loss or vessel enhancement.…”

Conditional Segmentation in Lieu of Image Registration

“…We compared the proposed conditional segmentation network with a previously-proposed weakly-supervised registration network [5], because 1) it uses the same types of image and ROI data in training; and 2) it was proposed with a clinical aim for predicting ROIs, including the prostate gland, one or more image-visible lesions (potentially tumours) and surrounding organs, so these can be identified during ultrasound-guided interventional procedures [5,[8][9][10]. Once trained, the registration network does not need the moving ROI as input to predict a DDF for each image pair.…”