We analyzed chemical composition, mineralogy, and spectral characteristics of the tailings of a hydrothermal gold mine in South Korea. We measured spectral responses of tailings to arsenic (As) and lead (Pb) concentration and developed and validated a prediction model for As and Pb in the tailings. The tailing was heavily contaminated with heavy metal elements and composed of rock forming minerals, gangue minerals and hydrothermal alteration minerals. The spectral features of the tailing were closely related to hydrothermal alteration minerals. The spectral responses associated with As and Pb concentrations were detected in shortwave infrared (SWIR) region at absorption positions of the hydrothermal alteration minerals. The prediction models were constructed using spectral bands of absorption features of the hydrothermal alteration minerals and were statistically significant. We found distinctive differences in spectral characteristics and spectral response to heavy metal contamination between the tailings and soils in the mining area. While the spectral signals to heavy metal concentration of tailings were associated with the hydrothermal alteration minerals, those of soils in mining area were manifested by clay minerals originated from weathering processes. This infers that geological processes associated with formation of soils and tailings are the major controlling factors of spectral responses to heavy metal contamination. This study provides a rare reference for the estimation of As and Pb concentration in the tailings with similar types of ore deposit and host rock.