Observations indicate that the Arctic sea ice cover is rapidly retreating while the Antarctic sea ice cover is steadily expanding. State-of-the-art climate models, by contrast, typically simulate a moderate decrease in both the Arctic and Antarctic sea ice covers. However, in each hemisphere there is a small subset of model simulations that have sea ice trends similar to the observations. Based on this, a number of recent studies have suggested that the models are consistent with the observations in each hemisphere when simulated internal climate variability is taken into account. Here we examine sea ice changes during 1979-2013 in simulations from the most recent Coupled Model Intercomparison Project (CMIP5) as well as the Community Earth System Model Large Ensemble (CESM-LE), drawing on previous work that found a close relationship in climate models between global-mean surface temperature and sea ice extent. We find that all of the simulations with 1979-2013 Arctic sea ice retreat as fast as observed have considerably more global warming than observations during this time period. Using two separate methods to estimate the sea ice retreat that would occur under the observed level of global warming in each simulation in both ensembles, we find that simulated Arctic sea ice retreat as fast as observed would occur less than 1% of the time. This implies that the models are not consistent with the observations. In the Antarctic, we find that simulated sea ice expansion as fast as observed typically corresponds with too little global warming, although these results are more equivocal. We show that because of this, the simulations do not capture the observed asymmetry between Arctic and Antarctic sea ice trends. This suggests that the models may be getting the right sea ice trends for the wrong reasons in both polar regions.