Background/Objectives: The Mediterranean dietary pattern (MedDiet) has numerous health benefits, particularly in preventing chronic diseases and improving well-being. Given the ageing population, understanding its impact on older adults’ physical health is essential. This study examines how adherence to the MedDiet influences the physical component (Comp-p) of health-related quality of life (HRQoL) across various age groups, providing insights for tailored dietary interventions. Methods: A cross-sectional study was conducted with active adults aged 41–80, categorised into four age groups (41–50, n = 116; 51–60, n = 225; 61–70, n = 135; 71–80, n = 44). Data were collected using the SF-36 and MEDAS questionnaires. Com-p scores were analysed based on MedDiet adherence (poor or good) and age. Results: In the 71–80 age group, a significant correlation was found between Comp-P and MedDiet adherence (r = 0.367, p = 0.014), with significantly higher Com-P scores in the good adherence group (50.10 ± 7.39) compared to the poor group (44.46 ± 7.73; p = 0.015; d = 0.74). The loss of adherence to the Mediterranean diet in this age group was attributed to low consumption of vegetables (36.36%), tree nuts (47.73%), legumes (50.00%), fish (52.27%), and fruit (56.82%). Conclusions: In individuals aged 71–80, lower adherence to the Mediterranean diet is associated with a decline in self-perceived physical health, attributed to the reduced intake of fresh vegetables, legumes, fish, and fruit. These findings emphasise the importance of promoting Mediterranean dietary adherence in later life to maintain optimal physical well-being.