Dietary exposure to trace metals (TMs) through vegetable consumption has been identified as a potential risk to human health. Fifty-one paired agricultural soil and leaf vegetable samples were collected around V-containing stone coal smelting sites in Hunan Province, China, to study the contamination and transfer characteristics of TMs (Cd, Cr, Cu, Pb, V, and Zn) in the soil–vegetable system. The health risk to local residents through vegetable ingestion was evaluated using Monte Carlo simulations. The results showed that 96.2%, 23.1%, 53.8%, 30.8%, 96.2%, and 69.2% of the soil samples had Cd, Cr, Cu, Pb, V, and Zn contents exceeding their related maximum allowable values, respectively. Cadmium and V were the primary pollutants based on the Igeo values. Moreover, 46.9% and 48.4% of vegetable samples exceeded the maximum permissible levels for Cd and Pb, respectively. There was a negative correlation between the bioaccumulation factors for Cd and V of the vegetable and soil physicochemical properties, including pH, organic matter, and free Fe2O3 content. Ingestion of garland chrysanthemum and pak choi poses high health risks, and Cd, V, and Pb were the primary contributors. These findings will help design strategies to minimize contamination and human exposure to soil–vegetable systems caused by V-containing stone coal smelting.