Rice with a black-colored pericarp (hereafter, black rice) has attracted considerable attention because of its high nutritional value. It is traditionally grown in infertile agroecosystems, where soil nutrient deficiency would affect grain yield and nutritional quality of the rice. This study examined the effects of phosphorus (P) application (0, 100, 250, 500, 1,000, and 1,300 mg P2O5 pot−1) on the growth, grain yield, and certain quality parameters of unpolished black (“Asamurasaki”) and white (“Takanari”) rice. The results showed that black rice grew poorly under P limitation, resulting in fewer tillers and a lower grain yield, while maintaining higher photosynthetic efficiency than white rice. However, black rice produced under P stress exhibited higher levels of Zn, Fe, and antioxidant (total phenolic and flavonoid content) than white rice, making it more nutritious. The optimal P application rates for black and white rice were 250 and 500 mg P2O5 pot−1, respectively, to achieve high yields. At the optimal rate, black rice exhibited high Fe, Zn and antioxidant levels compared to that at higher P application rates and had lower phytic acid levels, which could alleviate its inhibition of mineral absorption in the digestive tract. Moreover, low grain P due to P stress decreased the lightness L* value in the CIE-Lab system, improving the visual grain quality. Therefore, optimal P management is crucial for maintaining optimal grain yield with low phytic acid concentration and enhancing the nutritional quality of black rice for human consumption.