Mining activities cause surface sulfate enrichment, which has negative impacts on human health and ecosystems. These high concentrations of sulfate may enter groundwater through the unsaturated zone (UZ), threatening groundwater quality. Therefore, we combined hydrochemical and dual isotopic analyses of sulfate in surface water, soil water and groundwater with evaluations of the UZ to identify the groundwater sulfate source and transformation in the coal mining area. Soil profile samples were collected near gangue heaps (UZ−1, UZ−2) and the mean sulfate concentrations of the UZ−1 profile and UZ−2 profile were 35.4 mg/L and 69.63 mg/L, respectively. The shallow groundwater sulfate was mainly from dissolution of evaporite, sulfide oxidation and sewage. Different sulfate contaminated areas showed different characteristics of sulfate sources. The sulfate source to groundwater near the coal gangue heaps was sulfide oxidation. The groundwater sulfate near the gangue heaps and industrial park compound contamination area was mainly derived from industrial and domestic sewage and sulfide oxidation. In addition, the role of bacterial sulfate reduction (BSR) in the groundwater was not obvious. This research result is of great significance for promoting the safe mining of coal resources and sustainable utilization of groundwater in the Huaibei coal mining area and other coal mining areas in China.