The excessive use of chemical phosphorus (P) fertilizer can lower grain yield and P use efficiency (PUE) by aggravating soil acidity. Substituting organic manure for chemical fertilizer can alleviate the problem, although the long-term effect of organic manure incorporation is unknown. We conceptualized that substituting organic manure for chemical fertilizer may result in higher crop yields and PUE. Therefore, the impact of long-term fertilizer treatments: (i) CK (control), (ii) PK (phosphorus and potassium fertilizer), (iii) NP (nitrogen and P fertilizer), (iv) NK fertilizer, (v) NPK fertilizer, and (vi) NPKM (30% NPK fertilizer plus 70% manure) on rice yield, PUE, P uptake, and apparent balance (APB) was investigated. The results showed that rice yield under different fertilizer treatments ranged from 6.2 to 11.8 t ha−1 (1984–1995), 7.9 to 12.7 t ha−1 (1996–2007), and 6.6 to 12.8 t ha−1 (2008–2018). The rice yield under NPKM was greatly improved compared to other treatments, except with that of NPK (1984–1995). Soil organic carbon (SOC), available P and phosphorus activation coefficient (PAC) under NPKM were significantly higher than other treatments during 1984–2018. Soil pH (1984–2018) was greatly higher under CK and NPKM than under other treatments. Soil total P under PK, NP, NPK, and NPKM was significantly higher than under CK and NK (1984–2018). Compared to other treatments, P uptake was significantly higher under NPKM, except with that of NPK (1984–1995 and 2008–2018). The average PUE (1984–2018) was 10.7, 20.2, 36.1, and 44.2 kg kg−1 under PK, NP, NPK, and NPKM, respectively. The APB under NPKM was significantly lower as compared to PK, NP, and NPK treatments. Therefore, we conclude that in addition to improving soil organic carbon, cations inputs from organic manure can be a factor for the increase in soil pH, making organic manure substitution for chemical fertilizer a more efficient strategy for increasing PUE and crop yield.