Carbon sequestration is the primary function of biochar. Hence, it is necessary to design biochar with high carbon (C) retention and low C loss. In this study, three P compounds, including KH2PO4, Ca(H2PO4)2, and NH4H2PO4, were premixed with corn stalk (1:4, w/w), aiming to produce biochars (CSB+K, CSB+Ca, and CSB+N) with high C sequestration and slow release of P at three temperatures (300, 500, and 700 °C). The addition of all P sources obviously increased C retention, with the order of NH4H2PO4 (65.6–83.5%) > Ca(H2PO4)2 (60.4–78.2%) > KH2PO4 (50.1–76.1%), compared with the pristine biochar (47.8–73.6%). The addition of Ca(H2PO4)2 and KH2PO4 led to an increase in aromaticity and graphitization, as evidenced by H/C, FTIR, Raman and XPS analysis, whereas an opposite result occurred on CSB+N. Furthermore, all three phosphates reduced C loss of biochars with H2O2 oxidation, and CSB+Ca showed the best effect. Ca(H2PO4)2 and KH2PO4 pretreated biochars had higher resistance to K2Cr2O7 oxidation and thermal treatment. In contrast, the C loss of NH4H2PO4-added biochar at 500 and 700 °C with K2Cr2O7 oxidation was increased by 54% and 36%, respectively. During the pyrolysis process, Ca(H2PO4)2 was transformed into insoluble Ca2P2O7, leading to the lowest P release rate of CSB+Ca. This study indicates that co-pyrolysis of corn stalk and Ca(H2PO4)2 is optimal for increasing C retention, enhancing C stability and improving slow-release performance of P regardless of pyrolysis temperature.