Earth's climate can be understood as a dynamical system that changes due to external forcing and internal couplings. Essential climate variables, such as surface air temperature, describe this dynamics. Our current interglacial, the Holocene (11,700 years ago to today), has been characterized by great stability of global mean temperature prior to anthropogenic warming. However, the mechanisms and spatiotemporal patterns of fluctuations around this mean, called temperature variability, are poorly understood despite their socio-economic relevance for climate change mitigation and adaptation. Here, we examine discrepancies between temperature variability from model simulations and paleoclimate reconstructions by clarifying the stability of local and global surface air temperature on the timescale of years to centuries. To this end, we contrast power spectral densities (PSD) and their power-law scaling using simulated and observation-based temperature series of the last six thousand years. We further introduce the spectral gain to disentangle the externally-forced and internally-generated variability as a function of timescale. It is based on our estimate of the joint PSD of radiative forcing, which exhibits a scale break around the period of ten years. We find that local temperature series from paleoclimate reconstructions tend to show unstable behavior on periods of 10 to 200 years, while simulated temperatures almost exclusively show stable behavior. Conversely, the PSD and spectral gain of global mean temperature are consistent across data sets. Our results point to the limitation of climate models to fully represent local temperature statistics over decades to centuries. By highlighting the key characteristics of temperature variability, we pave a way to better constrain possible changes in temperature variability with global warming and assess future climate risks.