Concerns About Methodology in Study of Thyroid Nodule Gene Expression Classifier

Abstract

To the Editor Valderrabano et al 1 question the sensitivity and specificity of the Afirma gene expression classifier (GEC) from its pivotal clinical validation, and by extension, its negative predictive value (NPV). 2 I disagree with their methodology and conclusions.The authors report that among 1071 Afirma GEC benign results from 20 independent clinical experience publications, only 23 (2.1%) were reported as malignant on surgical pathologic examination (eTable 3). This supports Afirma's validated 94% or greater NPV, 2 which has allowed most Afirma benign nodules to undergo conservative clinical observation in lieu of costly and invasive diagnostic surgery. 3 The authors compare the relationship of benign call rate (BCR) and positive predictive value (PPV) in postmarketing studies with that predicted from validation to assess diagnostic performance generalizability. In validation, the BCR and PPV are appropriately measured in the same cohort of thyroid nodules that underwent surgical resection. 2 However, the authors, in extracting data from published studies, measured BCR in all tested thyroid nodules, yet measure PPV only among the 80% of Afirma GEC suspicious nodules that were resected (eTable3). This is a flaw in the analysis. Because resected Afirma suspicious nodules are more likely to have cancer than Afirma suspicious nodules that had not undergone resection, the observed PPV is overestimated and the observed PPV to BCR relationship predictably shifts to the right when compared with validation. The effect of uncoupling the BCR and PPV by measuring them in different thyroid nodule subsets is seen in Figure 3A-D, but wrongly attributed to Afirma GEC performance. This disconnection of BCR from observed PPV was not considered when interpreting Figure 4, where the authors report that a single cancer prevalence, applied to the validated diagnostic performance, could not explain the observed (postmarketing) BCR and PPV. This conundrum is not unexpected given the flawed methodology.In addition, the exclusion of most of the Afirma GEC benign nodules (because they are unresected) dramatically diminishes (and misrepresents) estimates of specificity and NPV throughout the article. 4 Unresected Afirma GEC benign nodules are highly likely to be benign, as has been shown in numerous studies. 3