Legumes have a high demand for phosphorus (P) but also have effective physiological and morphological strategies of P mobilisation. In order to evaluate the inter- and intraspecific P efficiency of small-grain legumes under contrasting long-term P management, eight accessions each of alfalfa (Medicago sativa L.) and red clover (Trifolium pratense L.) were cultivated in two consecutive growing periods from 2020 to 2021 in a field trial established in 1998. Six treatments (no P, triple-superphosphate (TSP), biomass ash, cattle manure, biowaste compost, and biowaste compost + TSP) were considered as P sources. While the yield clearly varied between both growing seasons, the differences between alfalfa and red clover were relatively small (4.7 vs. 4.9 Mg ha−1 in 2020 and 12.0 vs. 10.5 Mg ha−1 in 2021, p < 0.05). Even after more than 20 years of P management, crop yields were hardly affected by mineral P sources (TSP and biomass ash) while organic fertilisers increased the yields and nutrient uptake of plants and also raised soil P pools and the activities of soil enzymes in comparison to the control. A relevant crop effect was only found for the nitrogen (N) leaching with higher mineral N contents in 60 to 90 cm soil depth measured for red clover compared to alfalfa (11.8 vs. 4.8 kg ha−1, p < 0.05). Our results emphasise the high P efficiency of small-grain legumes without pronounced inter- or intraspecific differences. The yield-enhancing effect of organic amendments was related to higher soil fertility rather than to P supply.