Mushroom residues and swine manure are two common types of agricultural waste that are often returned to fields as organic fertilizers. However, the environmental risks of their reclamation, such as heavy metal pollution, have been less studied. To investigate the potential risks of heavy metal contamination in soils and vegetables after continuously applying mushroom residues and swine manure, field experiments of four consecutive vegetable rotations were conducted in the Qingpu District of Shanghai, Southeast China, from 2019 to 2021. The concentrations of heavy metals in soils continuously fertilized with mushroom residues and swine manure gradually increased. The organic matter content in the soils exhibited a significant correlation with the concentrations of Cu, Zn, Pb, and Cd (p < 0.01), suggesting that the increase in heavy metals is attributed to the use of organic waste. In particular, the application of swine manure increased Cu, Zn, Pb, and Cd concentrations in the soils by 118.3%, 54.9%, 57.6%, and 122.2%, respectively. Moreover, the application of organic waste raised the risk of the bioaccumulation of toxic metals, such as Cd, in vegetables. The Cd concentration was significantly and positively correlated with Zn in the edible parts of vegetables (p < 0.05). The recycling of swine manure more significantly enhanced Cd concentrations in the edible parts of green pepper (Capsicum annuum), eggplant (Solanum melongena), Brassica chinensis, and lettuce (Lactuca sativa), which were 2.53, 1.55, 1.66, and 1.62 times that of the non-fertilizer control (CK), respectively. Although the increase in heavy metals in the soils and vegetables was still mild when compared with the set thresholds of soil and food safety after the four vegetable rotations, the trend of increase in toxic heavy metals in the food chain with a continuous application of organic waste should be carefully considered.