Geographical accessibility to health care services is widely accepted as relevant to improve population health. However, measuring it is very complex, mainly when applied at administrative levels that go beyond the small-area level. This is the case in Portugal, where the municipality is the administrative level that is most appropriate for implementing policies to improve the access to those services. The aim of this paper is to assess whether inequalities in terms of access to a hospital in Portugal have improved over the last 20 years. A population-weighted driving time was applied using the census tract population, the roads network, the reference hospitals’ catchment area and the municipality boundaries. The results show that municipalities are 25 min away from the hospital—3 min less than in 1991—and that there is an association with premature mortality, elderly population and population density. However, disparities between municipalities are still huge. Municipalities with higher rates of older populations, isolated communities or those located closer to the border with Spain face harder challenges and require greater attention from local administration. Since municipalities now have responsibilities for health, it is important they implement interventions at the local level to tackle disparities impacting access to healthcare.