Deer populations and their impacts on forest ecosystems are increasing globally. Given the imperative and expense to mitigate impacts of invasive deer, we aimed to elucidate critical drivers of (i) deer density, (ii) deer impacts, and (iii) the relationship between them, to facilitate targeted management. We used quantile regression forests to model deer density (faecal pellet counts at 1948 locations) and impacts (browsing and other impacts on > 23,000 woody plants at 343 locations) across a mosaic of agricultural and forested ecosystems in Victoria, Australia (12,775 km2). Climate, topography, vegetation cover, and distance to water features were included as model covariates. Modelled deer density (r2 = 0.71, MAE = 0.56 pellets/m2) was most influenced by distance to waterbodies (> 10 ha, 31.2%), elevation (14.3%) and woody vegetation cover (12.9%). Modelled deer impact (r2 = 0.32, MAE = 6.9%) was most influenced by deer density (21.0%), mean annual precipitation (12.8%) and elevation (12.2%). Deer density was typically highest near large waterbodies, at low elevation, and with intermediate tree cover (40–70%). Impacts increased steadily with deer density up to ~ 2 pellets/m2. Our study demonstrates the importance of forest water and forest agricultural interfaces for both deer density and impacts. Deer are likely to be most abundant near waterbodies due to the availability of high-quality forage and water, and prefer lowland locations that have access to both open and forested habitats. Spatial models can be used to predict deer density and associated impacts to facilitate targeted invasive deer management.