Quantifying bias in a health survey: Modeling total survey error in the National Immunization Survey

Abstract

Random-digit-dial telephone surveys are experiencing both declining response rates and increasing under-coverage due to the prevalence of households that substitute a wireless telephone for their residential landline telephone. These changes increase the potential for bias in survey estimates and heighten the need for survey researchers to evaluate the sources and magnitudes of potential bias. We apply a Monte Carlo simulation-based approach to assess bias in the NIS, a land-line telephone survey of 19-35 month-old children used to obtain national vaccination coverage estimates. We develop a model describing the survey stages at which component nonsampling error may be introduced due to nonresponse and under-coverage. We use that model and components of error estimated in special studies to quantify the extent to which noncoverage and nonresponse may bias the vaccination coverage estimates obtained from the NIS and present a distribution of the total survey error. Results indicated that the total error followed a normal distribution with mean of 1.72 per cent(95 per cent CI: 1.71, 1.74 per cent) and final adjusted survey weights corrected for this error. Although small, the largest contributor to error in terms of magnitude was nonresponse of immunization providers. The total error was most sensitive to declines in coverage due to cell phone only households. These results indicate that, while response rates and coverage may be declining, total survey error is quite small. Since response rates have historically been used to proxy for total survey error, the finding that these rates do not accurately reflect bias is important for evaluation of survey data. Published in 2011 by John Wiley & Sons, Ltd.

Since The Start Of The Vaccines For Children Program, Uptake Has Increased, And Most Disparities Have Decreased

“…The accuracy of the estimates using these weights has been shown to allow for a total error of less than 1.72 percent from the true population estimate. [27] A total of 342,062 children ages 19- …”

Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research

“…Good examples of nonresponse bias analysis and data quality profiles are produced by several surveys. For example the National Immunization Survey's detailed total survey error analysis (Molinari et al 2011) and the Current Population Survey's data quality profile (US Census Bureau 2002). However, as the Federal Committee on Statistical Methodology wrote in 2002 (FCSM 2001) there is a lack of consistency within agencies, and between agencies, with respect to the types of evaluations being published on the survey errors for surveys that make up the federal statistical system (FCSM 2001).…”

The Association of Hepatitis B Vaccine Supply Policy with Timing of Receipt of the First Dose of Hepatitis B Vaccination

“…Finally, a few sub-state estimation areas rotated in or out in the survey design during the last few years. However a recent study [21] suggested that the total survey error in the NIS may be small with a mean of 1.7%, 95%CI (1.71%, 1.74%), and would likely not have changed the conclusions in this study.…”