Health-related quality of life (HRQoL) is influenced by several factors, such as living place, physical activity (PA), and functional fitness levels. Evidence shows that functional fitness and PA levels are strongly associated with positive HRQoL, especially in the older population. However, the impact of the living place has not been investigated as an influencing variable in this context. Therefore, this study aimed to investigate the relationship between the HRQoL, PA, and functional fitness of older adults living in rural and urban areas of Portugal. A cross-sectional study was performed with community-dwelling adults aged 65 years and over (n = 261) living in the city of Guarda. The participants were assessed for sociodemographic, anthropometric, clinical health, HRQoL, PA levels, and functional fitness status. The results showed that rural residents presented higher scores of HRQoL and functional fitness than older individuals living in urban areas. Regression models showed that functional fitness variables influence the HRQoL overall score and mental and physical subcomponents of HRQoL, regardless of the living place. In contrast, PA levels only influenced the HRQoL score in rural residents. The findings suggest that intervention programs to improve the physical health, quality of life, and well-being of the older population need to consider the country’s different geographical areas.