The rangelands of Crete island (Greece) are typical Mediterranean habitats under high risk of degradation due to long-term grazing and harsh climatic conditions. We explored the effect of abiotic (climatic conditions, altitude) and biotic factors (long-term grazing by small ruminants) on the floristic composition and diversity of selected lowland (Pyrathi, Faistos) and highland (Vroulidia, Nida) rangelands. In each rangeland, the ground cover was measured, and the floristic composition was calculated in terms of five functional groups: grasses, legumes, forbs, phrygana, and shrubs. The aridity index, species turnover, species richness, Shannon entropy, and Gini–Simpson index (with the latter two converted to the effective number of species) were calculated. Our results reveal that highlands are characterized by the highest aridity index (wetter conditions). Lowland rangelands, compared to highland, exhibited a higher percentage contribution of grasses, legumes, and forbs, while species turnover decreased along the altitudinal gradient. The Shannon entropy index was correlated (a) positively with Gini–Simpson and mean annual temperature and (b) negatively with mean annual precipitation, aridity index, and altitude. Moreover, the Gini–Simpson index correlated positively with mean annual temperature and negatively with altitude. Our results could help to understand the effects of grazing on rangeland dynamics and sustainability in semi-arid regions in the context of climatic change.