Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

Abstract

Quantitative imaging biomarkers (QIBs) are being used increasingly in medicine to diagnose and monitor patients' disease. The computer algorithms that measure QIBs have different technical performance characteristics. In this paper we illustrate the appropriate statistical methods for assessing and comparing the bias, precision, and agreement of computer algorithms. We use data from three studies of pulmonary nodules. The first study is a small phantom study used to illustrate metrics for assessing repeatability. The second study is a large phantom study allowing assessment of four algorithms' bias and reproducibility for measuring tumor volume and the change in tumor volume. The third study is a small clinical study of patients whose tumors were measured on two occasions. This study allows a direct assessment of six algorithms' performance for measuring tumor change. With these three examples we compare and contrast study designs and performance metrics, and we illustrate the advantages and limitations of various common statistical methods for QIB studies.

A Comparison of Lung Nodule Segmentation Algorithms: Methods and Results from a Multi-institutional Study

“…However, when normality is assumed, the CCC equals the ICC [34,36,37]. Although we report the CCC here, this measure does suffer from notable deficiencies common to many of these correlation measures [39][40][41] in that they are very sensitive to sample heterogeneity and that they are aggregate measures (thus making it difficult to separate systematic bias from issues in precision or large random errors). Thus, it would not be valid to compare our CCC measures to those measured on a different set of nodules with a different range of volumes.…”

“…Table 2 summarizes the metrics used in this study. For more details, readers are referred to a series of papers published on statistical methods for quantitative imaging biomarkers [31][32][33]41].…”

Metrology Standards for Quantitative Imaging Biomarkers

“…More detail is provided by Raunig et al (4). Investigators often want to compare the technical performance of two or more competing imaging procedures to assess the typical performance of the procedures, to identify the best procedure, to test the noninferiority of a procedure relative to a standard procedure, or to identify procedures that provide similar measurements (6). Table 7 summarizes some common research questions asked in QIB procedure comparison studies and possible study designs used with each.…”

“…Comparison of intraclass correlation coefficients estimated from groups of subjects sampled from different populations can be misleading because intraclass correlation coefficients are scaled relative to the subjects in the study sample; thus, comparisons based on different populations can be invalid (5,6). Within-subject coefficient of variance The within-subject coefficient of variance is the standard deviation of the replicate measures (within-subject standard deviation) divided by the mean.…”

“…Otherwise, comparisons of the measurement values with the reference values will produce significantly different results from comparisons with the true values. Simulation studies from Obuchowski et al (6) indicate that the intraclass correlation coefficient that reflects the concordance between the reference values and the SPECIAL REPORT: Metrology Standards for Quantitative Imaging Biomarkers Sullivan et al but sources for laboratory assays recommend at least five to seven levels (17). In phantom studies, these should be appropriately spaced over the measuring interval to adequately characterize the bias.…”