The temperature history of the Holocene is intensely debated, with both proxy seasonality effects and climate model sensitivity being proposed as possible reasons for the “Holocene temperature conundrum”. However, possible human impacts on terrestrial proxies are often not adequately considered, especially for the late Holocene. Based on a review of pollen-based and pollen-dominated temperature records on regional, continental and global scales, we propose that pollen may be a poor indicator of temperature change in late Holocene, mainly as a result of the impacts of intensified human activities on terrestrial vegetation. Thus, we suggest that possible human impacts on terrestrial Holocene temperature records should be carefully evaluated; and also that more Holocene temperature records are obtained from areas without significant direct human impacts, or that proxies that are insensitive to human impacts are used for late Holocene climatic reconstruction.