This study aimed to evaluate the association between adherence to the Mediterranean diet and the 20-year incidence of type II diabetes mellitus (T2DM) among adults from the ATTICA study. This study involved a prospective cohort of 3042 men and women recruited at baseline from the Attica region in Greece. Sociodemographic, anthropometric, lifestyle, and clinical characteristics were evaluated at baseline and follow-up examinations; adherence to the Mediterranean diet was assessed through the MedDietScore (range 0–55); four Mediterranean diet trajectories were identified (i.e., increasing, decreasing, and sustained high and sustained low adherence levels). For the present analysis, data from 2000 individuals with complete information were used (age 43 ± 13 years; 49% men). Over the 20-year period, 26.3% (95%CI 24.4%, 28.3%) of participants developed T2DM; men exhibited a 1.5-times higher incidence compared to women (p < 0.001). Individuals consistently close to the Mediterranean diet throughout the studied period had an improved glycemic and lipidemic profile (at baseline and at 10-y follow-up) (all p-values < 0.001) and showed a 21% reduction in their 20-year risk of developing T2DM compared to those who were consistently away (RR = 0.79, 95%CI 0.47, 0.86). A long-term adherence to the Mediterranean diet is protective against the onset of T2DM and, therefore, could be incorporated in public health actions for the prevention of the disease.