The glacial cycles of the Quaternary heavily impacted species through refugial contractions and post-glaciation expansions. Similarly, population demographic trajectories were intensely shaped by human pressures such as harvest and land use changes, particularly during the colonial-era. White-tailed and mule deer survived in different refugia during the last glacial maximum and were heavily impacted by the European colonisation as harvest and forestry practices drove white-tailed deer to extirpation in some regions, and severely reduced mule deer populations. Here, we analysed 73 re-sequenced deer genomes form across their North American range to understand the consequences of climatic and anthropogenic pressures on deer demographic history and adaptation. We found a strong signal of glacial-induced vicariance and demographic decline; notably, there is a severe decline in white-tailed deer effective population size (Ne) at the end of the Last Glacial Maximum. We found robust evidence for a colonial impact in the form of a recent and dramatic drop in Ne in all analysed populations, followed by a recovery in mainland white-tailed deer only. Historical census size and restocking data show a remarkable parallel to historical Ne estimates, while temporal Ne/Nc ratio shows patterns of conservation concern for mule deer. Signatures of selection highlight genes related to olfaction genes essential to deer biology and a cold receptor previously detected in woolly mammoth. We also detected numerous immune-genes that we surmise reflect the changing land-use patterns in North America. Our study provides a detailed picture of anthropogenic and climatic-induced decline in deer diversity, and clues to understanding the conservation concerns of mule deer and the incredible demographic recovery of white-tailed deer.