“How much realism is needed?” — the wrong question in silico imagers have been asking

Abstract

Purpose: To discuss the use of realism as a first approximation for assessing computational imaging methods. Methods: Although in silico methods are increasingly becoming promising surrogates to physical experimentation for various stages of device development, their acceptance remains challenging. Realism is often considered as a first approximation for assessing computational imaging methods. However, realism is subjective and does not always ensure that key features of the methodologies reflect relevant aspects of devices of interest to imaging scientists, regulators, and medical practitioners. Moreover, in some cases (e.g., in computerized image analysis applications where human interpretation is not needed) how realistic in silico images are is irrelevant and perhaps misleading. Results: I emphasize a divergence from this methodology by providing a rationale for evaluating in silico imaging methods and tools in an objective and measurable manner. Conclusions: Improved approaches for in silico imaging will lead to the rapid advancement and acceptance of computational techniques in medical imaging primarily but not limited to the regulatory evaluation of new imaging products. Published 2017. This article is a U.S. Government work and is in the public domain in the USA. The realism of computer-generated imagery as simulation of real-world images is often used as part of the development and testing of new in silico imaging methods. Many imaging scientists use realism of the synthetic images as a key success indicator for an in silico methodology. Typical studies that investigate realism use a human interrogator (image reader) to visually evaluate images using a discrete scale with a blind experimental paradigm. In this type of experiment, sometimes known as a "fool the radiologist" study design, the experiment is deemed successful if the performance of the radiologists is found to be at almost guessing levels for distinguishing synthetic versus real-world images. A typical example of this procedure is found in Shaheen et al. 1 where synthetic mammographic lesions were inserted into normal patient images to augment datasets for training and testing of image analysis software. The study consisted of inserting MRI-derived, spiculated lesions into projection images to obtain mammograms and into tomographic sequences of projection images reconstructed to form tomosynthesis datasets. The realism of these models was assessed by five radiologists using a receiver operating characteristic (ROC) study as compared to 54 real-world masses. The authors reported that humans were only marginally capable of distinguishing synthetic from real-world images with area-under-the-ROC curve averaged over all observers of 0.54 [95% confidence interval (0.50, 0.66)] for the 2D study and 0.67 [95% confidence interval (0.55, 0.79)] for tomosynthesis. The study concluded that the mass models were validated for the realism of their appearance for digital mammography and breast tomosynthesis and proposed the database of models as suitable f...

Organ doses evaluation for chest computed tomography procedures with TL dosimeters: Comparison with Monte Carlo simulations

“…In particular, it is essential to evaluate the image quality when aiming at protocol optimization. There are several studies reporting different tools to assess clinical image quality, [54][55][56][57] although on the other hand there are several studies showing that a radiologist tend to select images in which a given objective parameter (e.g., contrast resolution) is higher. 54 In the study proposed by Rehani 54 The overall reduction in absorbed organ dose with TCM adjustments is in good agreement with the literature, although TCM differs per CT scanner and protocol.…”

Performance evaluation of digital breast tomosynthesis systems: comparison of current virtual clinical trial methods

“…Receiver Operating Characteristic (ROC) analysis is applied to establish whether reader response for the real and simulated datasets is significantly different (Shaheen et al 2011, Elangovan et al 2017. This type of experiment can be referred to as a 'fool the radiologist' study (Badano 2017). An alternative is to assess realism in terms of the purpose for which the image has been generated rather than simply whether the image simply looks realistic, even to an expert observer (Badano 2017).…”

“…This type of experiment can be referred to as a 'fool the radiologist' study (Badano 2017). An alternative is to assess realism in terms of the purpose for which the image has been generated rather than simply whether the image simply looks realistic, even to an expert observer (Badano 2017). Badano suggests that 'functional realism' is required (Ferwerda 2003, Badano 2017, where an image is defined as realistic if it provides the same visual information as the original scene.…”

Advances in Computational Human Phantoms and Their Applications in Biomedical Engineering—A Topical Review

“…Validation of VCT models for imaging device efficacy have traditionally involved “fool the reader” type studies intended to characterize how difficult it is for an expert to subjectively distinguish a real image from a simulated one [204]. On the other hand, it has been suggested that validity and applicability of VCTs for regulatory use should hinge upon whether regulatory decisions based on the VCT are as good as, and more efficient than, decisions based upon data from currently used experimental methods [205]. Validation would thus involve input from the regulatory agency regarding how VCT data is used in the decision-making process.…”