Background: Household survey data are frequently used to estimate vaccination coverage - a key indicator for monitoring and guiding immunization programs - in low and middle-income countries. Surveys typically rely on documented evidence from home-based records (HBR) and/or maternal recall to determine a child’s vaccination history, and may also include health facility sources, BCG scars, and/or serological data. However, there is no gold standard source for vaccination history and the accuracy of existing sources has been called into question. Methods and Findings: We conducted a systematic review of literature published January 1, 1975 through December 11, 2017 that compared vaccination status at the child-level from at least two sources of vaccination history. 27 articles met inclusion criteria. The percentage point difference in coverage estimates varied substantially when comparing caregiver recall to HBRs (median: +1, range: -43 to +17), to health facility records (median: +5, range: -29 to +34) and to serology (median: -20, range: -32 to +2). Ranges were also wide comparing HBRs to facility-based records (median: +17, range: -61 to +21) and to serology (median: +2, range: -38 to +36). Across 10 studies comparing recall to HBRs, Kappa values exceeded 0.60 in 45% of comparisons; across 7 studies comparing recall to facility-based records, Kappa never reached 0.60. Agreement varied depending on study setting, coverage level, antigen type, number of doses, and child age. Conclusions: Recall and HBR provide relatively concordant vaccination histories in some settings, but both have poor agreement with facility-based records and serology. Long-term, improving clinical decision making and vaccination coverage estimates will depend on strengthening administrative systems and record keeping practices. Short-term, there must be greater recognition of imperfections across available vaccination history sources and explicit clarity regarding survey goals and the level of precision, potential biases, and associated resources needed to achieve these goals.

Background A common means of vaccination coverage measurement is the administrative method, done by dividing the aggregated number of doses administered over a set period (numerator) by the target population (denominator). To assess the quality of national target populations, we defined nine potential denominator data inconsistencies or flags that would warrant further exploration and examination of data reported by Member States to the World Health Organization (WHO) and UNICEF between 2000 and 2016. Methods and findings We used the denominator reported to calculate national coverage for BCG, a tuberculosis vaccine, and for the third dose of diphtheria-tetanus-pertussis-containing (DTP3) vaccines, usually live births (LB) and surviving infants (SI), respectively. Out of 2,565 possible reporting events (data points for countries using administrative coverage with the vaccine in the schedule and year) for BCG and 2,939 possible reporting events for DTP3, 194 and 274 reporting events were missing, respectively. Reported coverage exceeding 100% was seen in 11% of all reporting events for BCG and in 6% for DTP3. Of all year-to-year percent differences in reported denominators, 12% and 11% exceeded 10% for reported LB and SI, respectively. The implied infant mortality rate, based on the country’s reported LB and SI, would be negative in 9% of all reporting events i.e., the country reported more SI than LB for the same year. Overall, reported LB and SI tended to be lower than the UN Population Division 2017 estimates, which would lead to overestimation of coverage, but this difference seems to be decreasing over time. Other inconsistencies were identified using the nine proposed criteria. Conclusions Applying a set of criteria to assess reported target populations used to estimate administrative vaccination coverage can flag potential quality issues related to the national denominators and may be useful to help monitor ongoing efforts to improve the quality of vaccination coverage estimates.

Background: Household survey data are frequently used to estimate vaccination coverage - a key indicator for monitoring and guiding immunization programs - in low and middle-income countries. Surveys typically rely on documented evidence from home-based records (HBR) and/or maternal recall to determine a child’s vaccination history, and may also include health facility sources, BCG scars, and/or serological data. However, there is no gold standard source for vaccination history and the accuracy of existing sources has been called into question. Methods and Findings: We conducted a systematic review of peer-reviewed literature published January 1, 1957 through December 11, 2017 that compared vaccination status at the child-level from at least two sources of vaccination history. 27 articles met inclusion criteria. The percentage point difference in coverage estimates varied substantially when comparing caregiver recall to HBRs (median: +1, range: -43 to +17), to health facility records (median: +5, range: -29 to +34) and to serology (median: -20, range: -32 to +2). Ranges were also wide comparing HBRs to facility-based records (median: +17, range: -61 to +21) and to serology (median: +2, range: -38 to +36). Across 10 studies comparing recall to HBRs, Kappa values exceeded 0.60 in 45% of comparisons; across 7 studies comparing recall to facility-based records, Kappa never reached 0.60. Agreement varied depending on study setting, coverage level, antigen type, number of doses, and child age. Conclusions: Recall and HBR provide relatively concordant vaccination histories in some settings, but both have poor agreement with facility-based records and serology. Long-term, improving clinical decision making and vaccination coverage estimates will depend on strengthening administrative systems and record keeping practices. Short-term, there must be greater recognition of imperfections across available vaccination history sources and explicit clarity regarding survey goals and the level of precision, potential biases, and associated resources needed to achieve these goals.