Mining and smelting activities can contaminate soils and affect farming due to high emissions and input of potentially toxic elements (PTE) into the environment. Soils (sampled from two depths) and market vegetables from vegetable gardens located within the vicinity of unconfined slag deposits from decades of mining and smelting activities in Kutná Hora, Czechia were assessed to determine to what extent they pose a health hazard to communities that use these gardens. Pseudo-total As concentrations in the soils exceeded background levels (4.5 mg kg−1) 1.9–93 times, with higher concentrations in the deeper layer. The pseudo-total concentrations of PTE in soils ranked in the order As > Zn > Cd > Pb. Phyto-available concentrations of PTE in soils were relatively low, compared to pseudo-total concentrations. Concentration of As, Cd, Pb and Zn in the vegetables exceeded guideline values, with the highest concentrations found in the fruits of cucumber, peppers, and zucchini. Despite low phyto-available PTE concentrations in soils, all the PTE concentrations in the vegetables surpassed the guidelines set by the Czech Ministry of Health and EU directive, indicating a health hazard to consumers.