The accumulation/improper treatment of coal gangue will not only lead to waste of land, but also cause environmental pollution. Especially the impact of radioactive elements on the surrounding ecological environment is widely concerned by many scholars. In this study, the concentration of radioactive elements (uranium (U) and thorium (Th)) of small-scale coal gangue mining site and surrounding soil in the northern region of Xieqiao coal mine were tested, the material composition of coal gangue was analyzed via XRF and XRD, the modes of occurrence of U and Th elements were investigated, and their potential ecological risks and ecological effectiveness were evaluated. The results show that the clay minerals with high content in coal gangue are the key minerals for the adsorption of uranium and thorium in coal gangue. The specific activity of two radioactive elements (U and Th) in soil is much lower than that of coal gangue. With the increase of the distance from the soil collection point to the gangue piles and the depth of the soil profile, the specific activities of the two radioactive elements decrease gradually. On the basis of the concentration curve, the range of the radioactive contamination halo of gangue piles is limited (≤30 m), speculating qualitatively that the gangue dump has no significant influence on the radioactivity of the surrounding water. The modes of occurrence of U and Th in coal gangue and soil are altered. According to the index of geo-accumulation, Th is easier to accumulate in soil environment, but Th and U pollution in soil is not obvious. In contrast to U element, the active state of Th element in soil is generally affected by exogenous (coal gangue) export, which may have a potential environmental effects. This study provides a research idea for the investigation of radioactive element pollution to the surrounding soil in small-scale coal gangue plies.