Challenges to estimating vaccine impact using hospitalization data

Abstract

Because the real-world impact of new vaccines cannot be known before they are implemented in national programs, post-implementation studies at the population level are critical. Studies based on analysis of hospitalization rates of vaccine-preventable outcomes are typically used for this purpose. However, estimates of vaccine impact based on hospitalization data are particularly prone to confounding, as hospitalization rates are tightly linked to changes in the quality, access and use of the healthcare system, which often occur simultaneously with introduction of new vaccines. Here we illustrate how changes in healthcare delivery coincident with vaccine introduction can influence estimates of vaccine impact, using as an example reductions in infant pneumonia hospitalizations after introduction of the 10-valent pneumococcal conjugate vaccine (PCV10) in Brazil. To this end, we explore the effect of changes in several metrics of quality and access to public healthcare on trends in hospitalization rates before (2008–09) and after (2011–12) PCV10 introduction in 2010. Changes in infant pneumonia hospitalization rates following vaccine introduction were significantly associated with concomitant changes in hospital capacity and the fraction of the population using public hospitals. Importantly, reduction of pneumonia hospitalization rates after PCV10 were also associated with the expansion of outpatient services in several Brazilian states, falling more sharply where primary care coverage and the number of health units offering basic and emergency care increased more. We show that adjustments for unrelated (non-vaccine) trends commonly employed by impact studies, such as use of single control outcomes, are not always sufficient for accurate impact assessment. We discuss several ways to identify and overcome such biases, including sensitivity analyses using different denominators to calculate hospitalizations rates and methods that track changes in the outpatient setting. Employing these practices can improve the accuracy of vaccine impact estimates, particularly in evolving healthcare settings typical of low- and middle-income countries.

Population-level impact of infant 10-valent pneumococcal conjugate vaccination on adult pneumonia hospitalisations in Finland

“…In the USA, comparison of IPD rates reported by active surveillance with those estimated using ICD-coded data showed similar temporal trends 10. Changes in admission criteria might be associated with the observed changes in pneumonia hospitalisation rates, particularly if there was a shift to management of less severe cases in outpatient settings 32. The Finnish guidelines for the management of CAP published in 2008 introduced criteria for identifying low-risk patients who could be managed on outpatient basis 33.…”

Elucidating the impact of the pneumococcal conjugate vaccine programme on pneumonia, sepsis and otitis media hospital admissions in England using a composite control

“…Our method of assessing the impact of the PCV programme in England by comparing the IRR for the outcome of interest with that in control conditions has some advantages over other methods such as time-series analyses that have been used to assess the effect of PCV on pneumonia. The latter method does not take account of factors other than vaccination that may have resulted in secular changes in admission, and the results are sensitive to the model used to fit the pre-and post-intervention trend lines [ 33 ]. Another variation in the use of control conditions against which to assess the impact of PCV on pneumonia has recently been evaluated in a methodological study using hospitalisation data from five countries in the Americas [ 34 ].…”

Impact of Pneumococcal Conjugate Vaccines on Pneumonia Hospitalizations in High- and Low-Income Subpopulations in Brazil

“…In Brazil, comprehensive data are available on individuals who receive publicly funded healthcare; these accounted for approximately 82% of the national population in 2012, including >2.5 million infants aged <12 months, although the proportion covered varied locally [ 6 ]. We obtained deidentified, age-stratified, International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10), coded data on hospitalizations (January 2004–December 2013) from the Unified Health System (SIH-SUS, Ministry of Health).…”

“…The control variables were defined as in Bruhn et al [ 7 ]. The database, data quality issues, and case definitions have been described in detail previously [ 6 , 7 ].…”