Promise versus Reality: Optimism Bias in Package Inserts for Tuberculosis Diagnostics

Abstract

e Laboratorians and clinicians often rely on package inserts of diagnostic tests to assess their accuracy. We compared test accuracy for tuberculosis diagnostics reported in 19 package inserts against estimates in published meta-analyses and found that package inserts generally report overoptimistic accuracy estimates. However, package inserts of most tests approved by the U.S. Food and Drug Administration (FDA) or endorsed by the World Health Organization provide more realistic estimates that agree with meta-analyses. Most laboratory professionals anticipate that test performance when applied to patient care may be less impressive than what is reported in package inserts by test manufacturers. However, this gap between promise and reality has not been examined in a systematic way.One approach for studying this gap is to compare sensitivity and specificity estimates in package inserts with pooled estimates from published systematic reviews and meta-analyses, which often include studies from diverse settings and may provide a more realistic assessment of test accuracy.We conducted such a comparison using commercial tuberculosis (TB) diagnostics as a case study. The TB diagnostic pipeline has rapidly expanded, and a large number of meta-analyses have been published on various TB tests (24). Several tests are now endorsed by the World Health Organization (WHO). At the same time, there are examples of suboptimal and inaccurate TB diagnostic tests, and their use has been discouraged by the WHO (32). In fact, in vitro tests are often poorly regulated in many countries (13).We searched PubMed, the Cochrane Library, and the Evidence-Based TB Diagnosis website (www.tbevidence.org) for systematic reviews on the accuracy of diagnostics for TB published through March 2012 (search terms are available from the authors upon request). We excluded meta-analyses that reported only performance characteristics other than sensitivity and specificity (e.g., reproducibility, likelihood ratios), as they were not comparable to information provided in package inserts.We searched company websites for package inserts and contacted test manufacturers if package inserts were not available online. Diagnostic tests were not considered if they were not commercially available. Diagnostic tests for latent TB were not considered, because no good reference standard is available to determine accuracy.A total of 19 TB tests were included in the final analysis, because they met our eligibility criteria and package inserts as well as meta-analyses where available. These included gamma interferon (IFN-␥) release assays (IGRAs) for active TB, antibody-based serological tests, antigen detection tests, nucleic acid amplification tests (NAAT), and culture-based tests.As seen in Table 1, the quality of information provided in package inserts varies widely. Eighteen out of 19 package inserts in total overestimated the test accuracy. In particular, technologies that were neither recommended by the WHO nor approved by the U.S. Food and Drug Administration (FDA)...

Quantification of circulating Mycobacterium tuberculosis antigen peptides allows rapid diagnosis of active disease and treatment monitoring

“…Further reductions in operator time and assay cost, along with improvements in instrument portability, are needed to meet WHO guidelines for an optimal noninvasive TB assay. Larger, randomized prospective studies are warranted to confirm the results of this proof-of-principle pilot study, but a clinical method that shares the advantages of our approach should facilitate earlier interventions and better patient outcomes (44). The NanoDisk-MS assay also opens up new possibilities for the diagnosis of a wide variety of other infectious diseases, because it should be relatively simple to generate similar assays to rapidly quantify disease-associated low-abundance antigens in blood and other body fluids.…”

Challenges in the Development of an Immunochromatographic Interferon-Gamma Test for Diagnosis of Pleural Tuberculosis

“…The difference between findings in our study in comparison to the published literature on the performance of IFNg for the diagnosis of pleural TB could be reflective of both optimism bias and publication bias. Optimism bias for data reported in package inserts has been described for many TB tests [ 24 ], in particular those that did not undergo independent evaluation and approval by credible agencies such as the Federal Drug Administration (FDA) and World Health Organization (WHO). Most of the time this is observed when small numbers of patients are evaluated, controls are used that are not reflective of the population in regular clinical practice and high-risk groups are not included [ 24 ].…”

Current Status and Progress of Specific Laboratory Examination Methods of Active Tuberculosis Infection Diagnosis

“…Thus, the rapid quantitative detection of Mtbspecific proteins is very important. Both peptides are highly specific to Mtb [42], and the concentration of both peptides in serum is related to active TB infection [36,37]. Therefore, the determination of the free serum concentrations of these proteins can be used in the early diagnosis of Mtb.…”

“…Fortunately, these strains are a relatively rare form of TB; therefore, the potential for interference is minimal. Thus, the free concentrations of both protein peptides in serum can be used as an early signal of active TB infection [42]. To resolve the problem, tryptic peptides are selected to disrupt protein complexes in serum, releasing bound peptides.…”

“…So we do not need to overestimates the usefulness of the new method. The reliability of the Nanodisk-MS method needs further study [42].…”