SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments

Abstract

Artificial intelligence (AI) now enables automated interpretation of medical images for clinical use. However, AI's potential use for interventional images (versus those involved in triage or diagnosis), such as for guidance during surgery, remains largely untapped. This is because surgical AI systems are currently trained using post hoc analysis of data collected during live surgeries, which has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity, and a lack of ground truth. Here, we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization or adaptation techniques, results in models that on real data perform comparably to models trained on a precisely matched real data training set. Because synthetic generation of training data from human-based models scales easily, we find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real data-trained models due to the effectiveness of training on a larger dataset. We demonstrate the potential of SyntheX on three clinical tasks: Hip image analysis, surgical robotic tool detection, and COVID-19 lung lesion segmentation. SyntheX provides an opportunity to drastically accelerate the conception, design, and evaluation of intelligent systems for X-ray-based medicine. In addition, simulated image environments provide the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time, or mitigate human error, freed from the ethical and practical considerations of live human data collection.

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

“…Much of this review has focused on 2D imaging modalities for exactly this reason, since the generation of endoscopic, x-ray, or US images allows for hundreds or thousands of training samples to originate from a single patient model [44,97,161]. For example, DRRs vary widely in visual appearance based on the position and orientation of the virtual C-arm, and further techniques such as DR increase the variance of training data to further improve sim-to-real transfer [209,215,216]. However, existing techniques for generating these realistic-looking images rely on 3D patient models based on patient data, including CT, MRI, or prior endoscopic reconstructions, for example.…”

“…The advantages of DR are precisely proved in Gao et al [216], which shows that the combination of physics-based x-ray synthesis, using DeepDRR, combined with strong DR are comparable to GAN-based domain adaptation, and they outperform GAN-based domain adaptation with conventional DRRs, although this work is not yet peer-reviewed. This is advantageous because the image transformations involved in 'strong DR,' such as image inversion, blurring, warping, and coarse dropout, among others, are computationally inexpensive, whereas GANs require additional training with sufficient real images as an unlabeled reference.…”

“…This is advantageous because the image transformations involved in 'strong DR,' such as image inversion, blurring, warping, and coarse dropout, among others, are computationally inexpensive, whereas GANs require additional training with sufficient real images as an unlabeled reference. They demonstrate this approach, coined 'SyntheX,' on three representative tasks: pelvic landmark detection for 2D/3D registration, detection and segmentation of a continuum manipulator, and COVID-19 diagnosis from chest x-rays [216].…”