Phosphorus (P)-rich secondary raw materials can provide a valuable base for modern mineral fertilizers, provided that the new formulations do not load the soil–plant system with potentially toxic elements. Fertilizers from sewage sludge ash (SSA) and/or animal bones, activated by phosphorus-solubilizing bacteria (Bacillus megaterium or Acidithiobacillus ferrooxidans), were tested in field experiments in north-eastern Poland. The reference provided treatments with superphosphate and treatment without phosphorus fertilization. In one experiment, all P-fertilizers were applied at a P dose of 21 kg·ha−1, and in the other three experiments, three P doses were adopted: 17.6, 26.4, and 35.2 kg·ha−1. The effect of recycled fertilizers on the content of arsenic (As), chromium (Cr), nickel (Ni), copper (Cu), and zinc (Zn) in the soil, in wheat grain and straw (test plant), weeds, and post-harvest residues was investigated. The application of recycled fertilizers in P amounts up to 35.2 kg·ha−1 did not change the As, Cr, Ni, Cu, or Zn contents in the soil and plant biomass. The contents of these elements in soil were below the permissible levels for arable land in Poland. Their concentrations in wheat grain and straw did not exceed the permissible or suggested limits for plant material to be used for food and feed, while in the weed and post-harvest residue biomass, they usually fell within the biological plant variability ranges.