Climate change threatens the survival of coral reefs on a global scale, primarily through mass bleaching and mortality as a result of marine heatwaves. While these short-term effects are clear, predicting the fate of coral reefs over the coming century is a major challenge. One way to understand the longer-term effects of rapid climate change is to examine the response of coral populations to past climate shifts. Coastal and shallow-water marine ecosystems such as coral reefs have been reshaped many times by sea-level changes during the Pleistocene, yet, few studies have directly linked this with its consequences on population demographics, dispersal, and adaptation. Here we use powerful analytical techniques, afforded by haplotype phased whole-genomes, to establish such links for the reef-building coral, Acropora digitifera. We show that three genetically distinct populations are present in northwestern Australia, and that their rapid divergence since the last glacial maximum (LGM) can be explained by a combination of founder-effects and restricted gene flow. Signatures of selective sweeps, too strong to be explained by demographic history, are present in all three populations and overlap with genes that show different patterns of functional enrichment between inshore and offshore habitats. In contrast to rapid divergence in the host, we find that photosymbiont communities are largely undifferentiated between corals from all three locations, spanning almost 1000 km, indicating that selection on host genes and not acquisition of novel symbionts, has been the primary driver of adaptation for this species in northwestern Australia.