Synthesis and Edition of Ultrasound Images via Sketch Guided Progressive Growing GANS

Abstract

Ultrasound (US) is widely accepted in clinic for anatomical structure inspection. However, lacking in resources to practice US scan, novices often struggle to learn the operation skills. Also, in the deep learning era, automated US image analysis is limited by the lack of annotated samples. Efficiently synthesizing realistic, editable and high resolution US images can solve the problems. The task is challenging and previous methods can only partially complete it. In this paper, we devise a new framework for US image synthesis. Particularly, we firstly adopt a sketch generative adversarial networks (Sgan) to introduce background sketch upon object mask in a conditioned generative adversarial network. With enriched sketch cues, Sgan can generate realistic US images with editable and fine-grained structure details. Although effective, Sgan is hard to generate high resolution US images. To achieve this, we further implant the Sgan into a progressive growing scheme (PGSgan). By smoothly growing both generator and discriminator, PGSgan can gradually synthesize US images from low to high resolution. By synthesizing ovary and follicle US images, our extensive perceptual evaluation, user study and segmentation results prove the promising efficacy and efficiency of the proposed PGSgan.

Approaches and Limitations of Machine Learning for Synthetic Ultrasound Generation

“…After removing duplicates, 197 articles were screened to assess their eligibility. Subsequently, 19 studies fulfilled the proposed eligibility criteria and were included in this review 13–31 . A graphical representation of the screening process can be found in Figure 1.…”

“…The median size of the training datasets reported in the articles was 1565 images. Similar to conditional generation, preprocessing methods, such as cropping, scaling, flipping, and rotation, were used to augment the dataset; however, three studies failed to report any preprocessing performed on the training data 23,25,27 . Four studies reported the use of generative models previously proposed in the literature.…”

“…Bargsten et al 22 include a novel layer in the generator to assimilate the noise characteristic of US images. Liang et al 27 propose a novel architecture based on progressive growing GAN (PGGAN) to generate obstetric US; work that was later revised and improved in Liang et al 23 to include lung and infant hip joint US. Valanarasu et al 24 introduced self‐attention to the generation process to synthesize detailed brain US showing intraventricular hemorrhages.…”

Towards Clinical Application of Artificial Intelligence in Ultrasound Imaging

“…Deep attention networks have also been proposed for improved segmentation performance in US imaging, such as the attention-guided dual-path network [53] and a U-Net-based network combining a channel attention module and VGG [54]. A contrastive learning-based framework [55] and a framework based on the generative adversarial network (GAN) [56] with progressive learning have been reported to improve the boundary estimation in US imaging [57]. The critical issues resulting from the instability of the viewpoint and cross-section often become apparent when the clinical indexes are calculated using segmentation.…”

Digital Rock Reconstruction with User-Defined Properties Using Conditional Generative Adversarial Networks

“…To enable GANs to study images of different rock types simultaneously and enhance the representativeness of generated samples according to user-defined properties, we leverage the GANs in a conditioning manner, which was originally proposed by Mirza and Osindero (2014). In this study, we adopt progressively growing GANs as a basic architecture (Karras et al 2017), which has demonstrated excellent performance in image synthesis (Liang et al 2020;Wang et al 2018), and has been successfully applied in geological facies modeling (Song et al 2021). Inspired by the work of generating MNIST digits based on class labels with conditional GANs (Mirza and Osindero 2014), we make the rock type as one of the conditional information, aiming to generate samples with respect to a specified rock type.…”