Phosphorus (P) lost via leaching from agricultural land is of major concern for water resource managers worldwide, and colloidal phosphorus (CP) may have a high contribution, since it is an important mobile form of P in soil and subsurface drainage. The objective of this study is to relate P fertilization application rates to CP leaching. To eliminate the influence of climate and facilitate the accurate measurement of P contents in different soil layers, we established soil columns to investigate the impacts of fertilizer application rates and timing on P leaching. Therefore, a soil column leaching experiment was undertaken with different P fertilization application rates (0, 20, 40, 100, 200, and 400 mg kg−1) for purple soil in southwest China. P application rates had significant effects on CP and dissolved phosphorus concentrations in the top soils (p < 0.05) (e.g., 0–10 cm in this study), and they further increased P leaching loss by 24–375%. CP was the dominant P form and contributed 31–61% to total phosphorus in the leachate. The concentration of different P forms in leachates decreased significantly over time, and the risk of P leaching loss was greater within two weeks after P application (p < 0.05). The advisable range of P application rate is recommended to be 0–450 kg ha−1 for agricultural practice, and it is also recommended to keep P fertilizer in the soil for more than two weeks. Some countermeasures, related to application rates and timing, should be taken to minimize the buildup of P in the field and reduce the risk of P leaching.