To motivate contributions to public goods, should policy makers employ financial incentives like taxes, fines, subsidies, and rewards? While these are widely considered as the classic policy approach, a substantial academic literature suggests the impact of financial incentives is not always positive; they can sometimes fail or even backfire. To test whether policy makers are overly bullish about financial incentives, we asked county heads, mayors, and municipal government representatives of medium-to-large towns in Germany to predict the effects of a financial incentive on COVID-19 vaccination, and tested the exact same incentive in a field experiment involving all 41,548 inhabitants (clustered in 10,032 addresses) of the German town of Ravensburg. Whereas policy makers overwhelmingly predict that the financial incentive will increase vaccination—by 15.3 percentage points on average—the same financial incentive yielded a precisely estimated null effect on vaccination. We discuss when financial incentives are most likely to fail, and conclude that it is critical to educate policy makers on the potential pitfalls of employing financial incentives to promote contributions to public goods.