Synthetic data accelerates the development of generalizable learning-based algorithms for X-ray image analysis

Abstract

Artificial intelligence (AI) now enables automated interpretation of medical images. However, AI's potential use for interventional image analysis remains largely untapped. This is because the post hoc analysis of data collected during live procedures has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity and a lack of ground truth. Here we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization techniques, results in machine learning models that on real data perform comparably to models trained on a precisely matched real data training set. We find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real-data-trained models due to the effectiveness of training on a larger dataset. SyntheX provides an opportunity to markedly accelerate the conception, design and evaluation of X-ray-based intelligent systems. In addition, SyntheX provides the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time or mitigate human error, free from the ethical and practical considerations of live human data collection.Advances in robotics and artificial intelligence (AI) are bringing autonomous surgical systems closer to reality. However, developing the AI backbones of such systems currently depends on collecting training data during routine surgeries. This remains one of the largest barriers to widespread use of AI systems in interventional clinical settings, versus triage or diagnostic settings, as the acquisition and annotation of interventional data is time intensive and costly. Furthermore, while this approach can contribute to the automation or streamlining of existing surgical workflows, robotic and autonomous systems promise even more substantial advances: novel and super-human techniques that improve outcomes, save time or mitigate human error. This is perhaps the most exciting frontier of computer-assisted intervention research.Conventional approaches for curating data for AI development (that is, sourcing it retroactively from clinical practice) are insufficient for training AI models that benefit interventions that use novel instrumentation, different access points or more flexible imaging. This is because they are, by definition, incompatible with contemporary clinical practices and such data do not emerge from routine care. Furthermore, these novel systems are not readily approved, and thus not easily or quickly introduced into clinical practice. Ex vivo experimentation

A multiscale and multicriteria generative adversarial network to synthesize 1-dimensional turbulent fields

“…Stochastic generative models, such as the one presented in this work, allow the generation of large amounts of data with specific statistics, which are useful for validating methodologies of analysis [67] or feeding machine learning algorithms when real data are scarce [68,69]. Furthermore, generative models provide new insights into the studied process itself by characterizing how the model operates to produce the stochastic field [66].…”

Simulation‐driven training of vision transformers enables metal artifact reduction of highly truncated CBCT scans

“…Therefore, the DeepDRR framework 27 is used in this work, because it could simulate physics‐based radiographs by considering spectrum‐aware forward projection, beam hardening, scatter estimation, and noise injection 20 . In addition, a recent publication in Nature Machine Intelligence underscores the effectiveness of DeepDRR in the advancement of generalizable deep learning‐based algorithms for real x‐ray image analysis 28 . In the process of forward projection, the source‐to‐detector distance is $$1164\,\textrm {mm}$$ and the source‐to‐isocenter distance is $$700\,\textrm {mm}$$.…”

Toward perception-based anticipation of cortical breach during K-wire fixation of the pelvis

“…These are placed along the field of view with random location and orientation. We project digitally reconstructed radiographs (DRRs) from the patient CT and tool mesh models using a modified version of the DeepDRR simulator [26], which has been shown to support sim-to-real transfer for X-rays [8,11,15,15]. Using this version, we simultaneously obtain realistic X-ray tansmission images and corresponding projected segmentations for each organ or tool present, enabling synchronous dataset generation of 448×448 images at a rate of ∼ 4 images / s on an RTX 2080 Ti, using less than 4 GB of GPU memory.…”

“…X-ray imaging is a workhorse imaging modality for diagnostic and interventional healthcare. There is enormous opportunity for quantitative, comprehensive, and automated segmentation of X-ray images to accelerate research and development in precision medicine [3,4,8,14,16,25,26,29]. Prior efforts have contributed machine learning techniques for X-ray image analysis that perform well within a narrow scope, but fail to apply broadly to a large swath of possible uses.…”

“…Recent advances in simulation [16,26] and sim-to-real transfer [8,16,26] introduce the possibility of training FMs for X-ray image analysis in a fully supervised manner. This opens the door to a language-aligned FM in the Xray domain, since perfect knowledge of simulated objects allows for automatic annotation of mask descriptions.…”