This study aimed to measure lead (Pb) and cadmium (Cd) exposure levels in residents living near a zinc (Zn) smelter in Seokpo-myeon, Bonghwa-gun, South Korea, and identify factors affecting exposure. Residents aged ≥20 years living within 3 km and ≥30 km away from the smelter were classified as the exposure group (n = 549), and the control group (n = 265), respectively. Data were obtained through a questionnaire survey. Blood Pb levels in the exposure group (4.19 µg/dL) were higher than in the control group (2.70 µg/dL). The exposure group (1.32 µg/L) also had higher urinary Cd concentrations than the control group (0.80 µg/L). Male sex, older age, previous work at the smelter, smoking, and proximity to the smelter were associated with higher blood Pb levels on multivariate analysis; urinary Cd concentration was significantly higher in women, those who were older, those with experience of working in a Zn smelter or mine, those with proximity to the Zn smelter, and those who consumed locally grown vegetables. In conclusion, Zn smelters are major source of Pb and Cd pollution and require ongoing public health management to prevent potential adverse health effects.