Weather is one of the most basic factors impacting animal populations, but the typical strength of such impacts on population dynamics is unknown. We incorporate weather and climate index data into analysis of 492 time series of mammals, birds and insects from the global population dynamics database. A conundrum is that a multitude of weather data may a priori be considered potentially important and hence present a risk of statistical over-fitting. We find that model selection or averaging alone could spuriously indicate that weather provides strong improvements to short-term population prediction accuracy. However, a block randomization test reveals that most improvements result from over-fitting. Weather and climate variables do, in general, improve predictions, but improvements were barely detectable despite the large number of datasets considered. Climate indices such as North Atlantic Oscillation are not better predictors of population change than local weather variables. Insect time series are typically less predictable than bird or mammal time series, although all taxonomic classes display low predictability. Our results are in line with the view that population dynamics is often too complex to allow resolving mechanisms from time series, but we argue that time series analysis can still be useful for estimating net environmental effects.