Excess phosphorus (P) accumulation in the soil can change the bioavailability of P and increase the leaching risks, but the quantitative evaluation of these responses in acidic red soil is lacking. This study aimed to investigate the composition of soil P fractions under different phosphorus apparent balances (PAB) in acidic red soil and the bioavailability and the leaching change-points of different P fractions. Five phosphorus (P) fertilization rates were applied (0, 16.38, 32.75, 65.50, 131.00 kg P·ha−1) in every sweet corn cultivation from the field experiment, and the treatments were marked as P0, P1, P2, P3, and P4, respectively. The PAB showed negative values in P0 and P1 which were −49.0 and −15.0 kg P·ha–1 in two years, respectively. In contrast, PAB in P2 as well as in P3 and P4 were positive, the content ranging from 40.2 to 424.3 kg P·ha−1 in two years. Per 100 kg ha−1 P accumulate in the soil, the total P increased by 44.36 and 10.41 mg kg−1 in the surface (0–20 cm) and subsurface (20–40 cm) soil, respectively. The content of inorganic P fractions, including solution phosphate (Sol-P), aluminum phosphate (Al-P), iron phosphate (Fe-P), reduction phosphate (Red-P), and calcium phosphate (Ca-P), significantly increased by 0.25, 16.22, 22.08, 2.04, and 5.08 mg kg−1, respectively, in surface soil per 100 kg ha−1 P accumulated in the soil. Path analysis showed that the most important soil P fractions contributing to Olsen-P were Sol-P and Al-P, which can directly affect Olsen-P, and their coefficients were 0.24 and 0.73, respectively. Furthermore, the incubation experiments were conducted in the laboratory to investigate the leaching risk of different P fractions, and they showed Sol-P was a potential source of leaching, and the leaching change-points of Al-P and Fe-P were 74.70 and 78.34 mg·kg–1, respectively. Continuous P that accumulated in soil changed the composition of P fractions, and the bioavailability as well as the leaching risks increased. This is important in optimizing soil P fertilization management in agricultural ecosystems based on the bioavailability and critical levels for leaching of P fractions.