This paper examines the synthetic control method in contrast to commonly used difference‐in‐differences (DiD) estimation, in the context of a re‐evaluation of a pay‐for‐performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re‐weighting the control group so that it has similar pre‐intervention characteristics to the treated group.We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re‐analyse a recent study evaluating the effects of a hospital P4P scheme on risk‐adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non‐incentivised conditions. This result was robust to alternative specifications of the synthetic control method. © 2015 The Authors. Health Economics published by John Wiley & Sons Ltd.

The introduction of pay for performance in all NHS hospitals in one region of England was associated with a clinically significant reduction in mortality. As compared with a similar U.S. program, the U.K. program had larger bonuses and a greater investment by hospitals in quality-improvement activities. Further research is needed on how implementation of pay-for-performance programs influences their effects. (Funded by the NHS National Institute for Health Research.).

Despite little evidence of effectiveness, pay-for-performance programs are being adopted with the intent of improving the quality of care. The few studies evaluating these programs have shown only modest and short-term effects on hospital processes of care and even weaker evidence for effects on patient outcomes. In 2008, the Advancing Quality program was introduced in all 24 National Health Service hospitals in the northwest region of England that provided emergency care. Patient-level data were used to analyze patient mortality from all hospitals across England for 3 conditions included in the program and 6 conditions not included in the program for 18 months before and 18 months after the introduction of the program.The Advancing Quality program was the first hospital-based pay-for-performance program to be introduced in England. Hospitals were required to collect and submit data on 28 quality measures covering acute myocardial infarction, coronary artery bypass grafting, heart failure, hip and knee surgery, and pneumonia. At the end of the first year, hospitals that reported quality scores in the top and second quartiles received bonuses. For the next 6 months, the reward system changed so that bonuses could be earned on the basis of 3 criteria, attainment, improvement, and achievement, with their performance during the second year compared with the first year. Thereafter, the program was included in a pay-for-performance program that applied across all of England and involved withholding of payments rather than bonuses. Patient-level data were obtained for all patients treated for 1 of 3 conditions included in the program: acute myocardial infarction, heart failure, and pneumonia. All deaths that occurred within 30 days after admission were included in the analysis. Equivalent data were obtained for patients admitted for 6 diagnoses not included in the program (acute renal failure, alcoholic liver disease, intracranial injury, paralytic ileus and intestinal obstruction without hernia, pulmonary embolism, and duodenal ulcer). The 3-year period (2007Y2010) included 18 months before the program's introduction and the first 18 months of its operation. The final sample included 410,384 patients with pneumonia, 201,003 patients with heart failure, 245,187 patients with acute myocardial infarction, and 241,009 patients with conditions not included in the program. Between-region and within-region difference-in-differences analyses were used to compare changes in mortality over time between the northwest region and the rest of England for conditions included or not included in the program. Each analysis included an interaction term between the intervention group and the period after the implementation.For all conditions, patients in the northwest region were slightly younger but had more coexisting conditions. Similar changes over time in patient volumes and patient characteristics were observed in both areas. Risk-adjusted mortality for all the conditions decreased over the 3 years in the northwest region and the re...