The Southern Ocean is warming more rapidly than other parts of our planet. How this regions endemic biodiversity will respond to such changes can be illuminated by studying past events, through genetic analyses of time-series data sets including historic and fossil remains. Archaeological and subfossil remains show that the southern elephant seal (Mirounga leonina) was common along the coasts of Australia and New Zealand in the recent past. This species is now mostly confined to sub-Antarctic islands and the southern tip of South America. We analysed ancient seal samples from Australia (Tasmania), New Zealand, and the Antarctic mainland to examine how southern elephant seals have responded to a changing climate and anthropogenic pressures during the Holocene. Our analyses show that these seals formed part of a broader Australasian lineage, comprising seals from all sampled locations from the south Pacific sector of the Southern Ocean. Our study demonstrates that southern elephant seal populations have dynamically altered both range and population sizes under climatic and human pressures, over surprisingly short evolutionary timeframes for such a large, long-lived mammal.