Differentiating between analytical and diagnostic performance evaluation with a focus on the method comparison study and identification of bias

Abstract

Prior to introduction of a new method to the diagnostic laboratory, analytical performance must be validated to ensure operation within the manufacturer's specifications and/or within predetermined quality requirements. In addition, the new method may require diagnostic performance assessment to ensure it differentiates between diseased and nondiseased individuals as intended. These 2 phases of assessment, while complementary, are not equivalent and require a different set of experiments, statistical analyses, and interpretation. Studies of analytical performance typically include a method comparison experiment, the purpose of which is to identify bias (inaccuracy) of the "test" (or "index") method (new method) relative to a "comparative method" (established method). Analysis of method comparison data is facilitated by commercial software programs that present the statistical significance of identified bias; however, the clinical relevance of any bias also should be considered. Studies of diagnostic performance should not be pursued until analytical performance is fully characterized and may not be required for well-established tests or for those for which results are nonspecific (ie, not referable to a specific disease or condition). Diagnostic performance assessment may include assessment of sensitivity, specificity, predictive values, odds ratios, and/or likelihood ratios. The purpose of this review is to clarify differences between the assessment of analytical and diagnostic performance, and to explore the method comparison study and bias assessment from a perspective not addressed in prior veterinary articles.

Validation of an enzyme-linked immunosorbent assay for measurement of feline haptoglobin

“…Comparing 2 field methods, bias is expected and comprised of analytical error of both methods. 12 The origin of the unacceptable bias between the methods was not determined in the present study and requires further investigation. The existence of an analytical error resulting from dilution of the samples in the spectrophotometric assay is still not completely examined.…”

“…12 In the second phase of validation, acceptable diagnostic performance was indicated by the significantly higher concentrations of haptoglobin (P < 0.005) measured in the group of cats with systemic inflammation. Based on haptoglobin concentrations measured in the ELISA, cats with marked acute phase reactions could be differentiated from healthy cats and diseased cats without systemic inflammation as already demonstrated in dogs.…”

“…The ELISA was considered valid for the assessment of the acute phase response if the overlap performance of haptoglobin measured by the ELISA was able to discriminate cats with systemic inflammatory disease from clinically healthy cats and diseased cats without systemic inflammation. 12,20 Measurements below the LOD of the ELISA were set at 0.1 g/L. P values <0.05 were considered significant.…”

Comparison of 2 assays for measuring serum total thyroxine concentration in dogs and cats

“…The method‐comparison analysis was performed as recommended in ASVCP guidelines using commercial statistical software (Stata SE, version 14.1, College Station, Texas; MedCalc version 16.8.4, Ostend, Belgium) on the whole data set. Differences between analytic results for paired methods were calculated as raw difference and percentage deviation from the MTT4 concentration and are not reported.…”

Comparison of a full-spectrum multi-analyte clinical analyser with six reference instruments using canine and feline blood samples

“…In addition, a good agreement in the Bland-Altman difference plot was regarded as a percentage of non-outliers of the limits of agreement (LOA) of greater than 95%; conversely, a percentage of non-outliers of less than 95% was regarded as poor agreement (Jensen and Bantz 1993;Gary et al 1999;Jensen Kjelgaard-Hansen 2006;Bland and Altman 2010;Flatland et al 2014). The percentage of non-outliers of the LOA was defined as the percentage of data points not outside the LOA in the Bland-Altman difference plots.…”