Meta-analysis of the technical performance of an imaging procedure: Guidelines and statistical methodology

Abstract

Medical imaging serves many roles in patient care and the drug approval process, including assessing treatment response and guiding treatment decisions. These roles often involve a quantitative imaging biomarker, an objectively measured characteristic of the underlying anatomic structure or biochemical process derived from medical images. Before a quantitative imaging biomarker is accepted for use in such roles, the imaging procedure to acquire it must undergo evaluation of its technical performance, which entails assessment of performance metrics such as repeatability and reproducibility of the quantitative imaging biomarker. Ideally, this evaluation will involve quantitative summaries of results from multiple studies to overcome limitations due to the typically small sample sizes of technical performance studies and/or to include a broader range of clinical settings and patient populations. This paper is a review of meta-analysis procedures for such an evaluation, including identification of suitable studies, statistical methodology to evaluate and summarize the performance metrics, and complete and transparent reporting of the results. This review addresses challenges typical of meta-analyses of technical performance, particularly small study sizes, which often causes violations of assumptions underlying standard meta-analysis techniques. Alternative approaches to address these difficulties are also presented; simulation studies indicate that they outperform standard techniques when some studies are small. The meta-analysis procedures presented are also applied to actual [18F]-fluorodeoxyglucose positron emission tomography (FDG-PET) test–retest repeatability data for illustrative purposes.

Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

“…To construct a CI for the true change in the tumor’s volume, the linearity assumption must hold and we must know the value of the coefficient b in the relationship: Y(t) = a + bX(t) + ε(t) [5]. From the QIBA 3a study the point estimate of b is 0.93 from a simple least-squares linear regression; however, we recognize that a better approach to estimating b is through a meta-analysis of existing literature [21]. For illustration purposes, let b =0.93.…”

Imaging biomarker roadmap for cancer studies

“…Questions of how IB acquisition and analysis should be standardized, and how terminology should be harmonized have been addressed by numerous academic, clinical, industrial and regulatory groups. These groups include the FDA2,24, the US National Cancer Institute (NCI) through the Quantitative Imaging Network (QIN)25 and the Cancer Imaging Program phase I and II Imaging trials initiative26, the Quantitative Imaging Biomarkers Alliance (QIBA)27,28, the American College of Radiology Imaging Network (ACRIN)29, the European Society of Radiology (ESR)30, the European Organisation for Research and Treatment of Cancer (EORTC) through the QuIC-ConCePT consortium13,31, the European Association of Nuclear Medicine (EANM)32, the International Society for Magnetic Resonance in Medicine (ISMRM)33 and Cancer Research UK (CRUK)34. Their efforts have produced consensus guidelines for the acquisition and analysis of several IBs33–37.…”

Statistical Issues in Testing Conformance with the Quantitative Imaging Biomarker Alliance (QIBA) Profile Claims

“…This set might be the 95% confidence interval (CI) of the performance from a meta-analysis of published studies using a variety of imaging vendors under relevant conditions [6]. Alternatively, this set might be based on results from groundwork projects in QIBA [8] or conducted by another outside group.…”

“…Through a large volunteer effort, two workshops sponsored by QIBA were conducted to develop standard statistical methods for defining technical performance metrics for QIBs [2–6]. These statistical methods provide the framework for comparing technical performances of imaging procedures in the groundwork studies and for characterizing the performance in the Profile claims.…”