Analytic Validation of the Automated Bone Scan Index as an Imaging Biomarker to Standardize Quantitative Changes in Bone Scans of Patients with Metastatic Prostate Cancer

Abstract

A reproducible and quantitative imaging biomarker is needed to standardize the evaluation of changes in bone scans of prostate cancer patients with skeletal metastasis. We performed a series of analytic validation studies to evaluate the performance of the automated bone scan index (BSI) as an imaging biomarker in patients with metastatic prostate cancer. Methods Three separate analytic studies were performed to evaluate the accuracy, precision, and reproducibility of the automated BSI. Simulation study: bone scan simulations with predefined tumor burdens were created to assess accuracy and precision. Fifty bone scans were simulated with a tumor burden ranging from low to high disease confluence (0.10–13.0 BSI). A second group of 50 scans was divided into 5 subgroups, each containing 10 simulated bone scans, corresponding to BSI values of 0.5, 1.0, 3.0, 5.0, and 10.0. Repeat bone scan study: to assess the reproducibility in a routine clinical setting, 2 repeat bone scans were obtained from metastatic prostate cancer patients after a single 600-MBq 99mTc-methylene diphosphonate injection. Follow-up bone scan study: 2 follow-up bone scans of metastatic prostate cancer patients were analyzed to determine the interobserver variability between the automated BSIs and the visual interpretations in assessing changes. The automated BSI was generated using the upgraded EXINI boneBSI software (version 2). The results were evaluated using linear regression, Pearson correlation, Cohen κ measurement, coefficient of variation, and SD. Results Linearity of the automated BSI interpretations in the range of 0.10–13.0 was confirmed, and Pearson correlation was observed at 0.995 (n = 50; 95% confidence interval, 0.99–0.99; P < 0.0001). The mean coefficient of variation was less than 20%. The mean BSI difference between the 2 repeat bone scans of 35 patients was 0.05 (SD = 0.15), with an upper confidence limit of 0.30. The interobserver agreement in the automated BSI interpretations was more consistent (κ = 0.96, P < 0.0001) than the qualitative visual assessment of the changes (κ = 0.70, P < 0.0001) was in the bone scans of 173 patients. Conclusion The automated BSI provides a consistent imaging biomarker capable of standardizing quantitative changes in the bone scans of patients with metastatic prostate cancer.

A Preanalytic Validation Study of Automated Bone Scan Index: Effect on Accuracy and Reproducibility Due to the Procedural Variabilities in Bone Scan Image Acquisition

“…Additionally, the centralized visual assessment of a bone scan, in a multisite study, does not account for the deviations from the bone scan procedure guidelines. Recently, in the analytic validation study of automated BSI, we have demonstrated that with minimal manual supervision the automated BSI can minimize the interoperator variability in assessing change in bone scans (7). In this preanalytical study, we have identified scanning speed and the resulting image counts as a procedural factor in bone scan image acquisition that can affect the reproducibility and accuracy of automated BSI assessment and therefore needs subsequent standardization for prospective clinical studies.…”

“…With the computer automation, the time of detecting metastatic lesion and calculating the BSI was reduced to 3-5 s per scan. The recent analytic validation study of automated BSI demonstrated that the platform has consistent accuracy in low-to high-burden disease and that with minimal manual supervision the automated BSI can minimize the interoperator variability (6,7). The analytic validation study was a vital step forward in the continual effort to validate automated BSI as an imaging biomarker in patients with metastatic prostate cancer.…”

Bone scan index: A new biomarker of bone metastasis in patients with prostate cancer

“…The understanding of BSI ranges and bone imaging patterns are convenient to standardize the perception of disease severity and therapeutic changes when patients are referred to or from other hospitals. Analytical validation with a revised algorithm showed the linearity of automated BSI with simulated phantom BSI including ranges of BSI >10% …”

“…Analytical validation with a revised algorithm showed the linearity of automated BSI with simulated phantom BSI including ranges of BSI >10%. 15…”

Reply: A Common Mistake in Assessing the Diagnostic Value of a Test: Failure to Account for Statistical and Methodologic Issues

“…Since the automated bone scan index (BSI) is a quantitative parameter, the comments on qualitative assessments are inapplicable. Our analytic validation study demonstrated the validity of the automated BSI with a Pearson r of 0.99 (P , 0.0001) and with associated parameters of linear regression (slope, 0.80 [95% confidence interval, 0.78-0.83]; intercept, 0.38 [95% confidence interval, 0.25-0.51]) (2). In this study, Cohen k-agreement and other well-known standard tests for imaging biomarkers (3) were also used to evaluate the reliability of the automated BSI platform.…”