Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Abstract

Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

“…We empirically observed that, in the constrained version of 𝜂, the model translated background features on the tools, as it was observed in [14] and [20], while this did not happen in the unconstrained version of 𝐿 𝑆𝑆 . An example of how our model behaves with and without this constraint is illustrated in Fig.…”

“…Training a conditional GAN (cGAN) [11] using segmentation labels as the network's input [12] has been shown to produce high quality laparoscopic images, while a similar approach using robot kinematic information along with unsupervised I2I has also been reported [19]. Recently, a Cycle-GAN was used to generate synthetic tools from metallic textures for binary surgical tool segmentation [20]. This, however, can generate only rigid and non-articulated tools and still requires labelled real samples to support I2I.…”

“…Moreover, we chose not to include [28] in our comparison experiment since [9] showed to already outperform this method in terms of image quality and segmentation accuracy on the considered datasets. [20] was not considered as well as it is constrained to generate surgical images with rigid tools, while we are investigating articulated ones. Since CoSegGAN's original formulation involves using a validation set to choose the best model, we used 20% of test sets from each dataset to fulfil this requirement.…”

Estimation with Uncertainty via Conditional Generative Adversarial Networks

“…While typical GAN models produce random samples, conditional variants of GANs (cGANs) have been introduced to generate the desired samples and have shown fine results to produce samples by using conditional inputs [19][20][21][22]. Moreover, by taking advantage of such a innovative framework of GAN, the modified GAN models have been introduced in many applications [23][24][25][26][27][28][29].…”

Impact on Inference Model Performance for ML Tasks Using Real-Life Training Data and Synthetic Training Data from GANs

“…A way to address these issues can be the generation of synthetic training data, a field of research that has garnered increasing interest in past years [4,6,7]. Approaches reach from physics-based modeling techniques as in [8] over classic image augmentation techniques, e.g., [6], to Deep Learning-based modeling as for example in [9][10][11].…”