Neural Style Transfer Improves 3D Cardiovascular MR Image Segmentation on Inconsistent Data

Abstract

Three-dimensional medical image segmentation is one of the most important problems in medical image analysis and plays a key role in downstream diagnosis and treatment. Recent years, deep neural networks have made groundbreaking success in medical image segmentation problem. However, due to the high variance in instrumental parameters, experimental protocols, and subject appearances, the generalization of deep learning models is often hindered by the inconsistency in medical images generated by different machines and hospitals. In this work, we present StyleSegor, an efficient and easy-to-use strategy to alleviate this inconsistency issue. Specifically, neural style transfer algorithm is applied to unlabeled data in order to minimize the differences in image properties including brightness, contrast, texture, etc. between the labeled and unlabeled data. We also apply probabilistic adjustment on the network output and integrate multiple predictions through ensemble learning. On a publicly available whole heart segmentation benchmarking dataset from MICCAI HVSMR 2016 challenge, we have demonstrated an elevated dice accuracy surpassing current stateof-the-art method and notably, an improvement of the total score by 29.91%. StyleSegor is thus corroborated to be an accurate tool for 3D whole heart segmentation especially on highly inconsistent data, and is available at https://github.com/horsepurve/StyleSegor.

Making Radiomics More Reproducible across Scanner and Imaging Protocol Variations: A Review of Harmonization Methods

“…In case GANs were to be used for NST, the developer should be mindful of limitations that GANs pose, as discussed in the previous section. Many more such studies [121][122][123][124][125][126] were conducted by applying style transfer ods on medical images, and this approach has the potential to harmonize images, by image-to-image translations or domain transformations. Depending on the arc tures used to perform style transfer, paired or unpaired images might be needed, e harmonization is to be performed using cycleGAN or StyleGAN as a baseline then p images are not a requirement.…”

Domain generalization in deep learning for contrast-enhanced imaging

“…Another strategy investigated in this work to improve the scalability of single-center models consisted of applying the so-called transfer learning paradigm, by fine-tuning specific layers of the neural network with a reduced number of LGE-MRI images from the new clinical site (Method 3 in Figure 2). The approach has shown promise for multi-center image segmentation in cardiac cine-MRI [18], but is yet to be demonstrated for multi-center LGE-MRI imaging, where there is increased variability. The following steps are implemented in this work:…”

Remove Appearance Shift for Ultrasound Image Segmentation via Fast and Universal Style Transfer

“…Recently, ST has been proposed to remove the gap by rendering the appearance of unknown content images as style images. Previous ST-based methods [10,11] trained two models and developed two-stage systems for segmenting images with appearance shift. In the first stage, the content image is transferred into a stylized image by the ST model.…”

“…We compared them with the solution replacing DIN with AdaIN [14], and got two variants, S-AdaINSeg and M-AdaINSeg. We also implemented two typical two-stage frameworks for comparison, including the StyleSegor [10] and WaveCT-AIN (WCT-AIN) [11]. GAN based DA methods are not considered for comparison in this work, since they need samples from Ven-B and Ven-C for retraining.…”

“…Compared to DA, ST is independent on the target domain, retraining-free, and suitable for images with unknown appearance shift. Ma et al [10] made the early attempt to exploit an online ST to reduce the appearance variation for better cardiac MR segmentation. Liu et al [11] proposed an Adaptive Instance Normalization (AdaIN) based style transfer module for vendor adaptation.…”