The present study used soils collected from a small-scale gold mine area to determine the health risks due to trace elements to the at-risk population in the study area. The work involved 74 soil samples from four sampling categories: 29 samples were from the mining pits (MD), 18 samples from the first washing area (WA), 17 samples from the second washing area (WB), and 10 samples from the control area (C). All samples were analyzed for Cr, Cu, As, Pb, Cd, Co, Ni, Zn, and Hg using the Energy Dispersive X-Ray Florescence (ED-XRF) method. Trace element levels were found to vary across the four sampling categories. The concentrations of trace elements recorded from different sampling categories varied in an increasing order of MD > WA > WB > C. Mercury was detected in the highest levels (max. 3.72 ± 0.15) at WB while it was not detected in the samples from C. Samples from MD indicated that Cu (max. 737.66 ± 1.3 mg/kg) was found in the highest levels whereas Hg (mean = 0.007 mg/kg) was the lowest. At WA, Cu (max. = 178.97 ± 2.46 mg/kg) registered the highest average concentration while Hg (mean = 0.05 mg/kg) had the lowest concentration. For WB, Cu (max. = 230.66 ± 3.99 mg/kg) was found in the highest concentration. The hazard index value for all exposure routes was found to be 1.77, making noncarcinogenic effects significant to the adult population. For children, the hazard index value was 9.11, showing a severe noncarcinogenic effect on children living in the study area. For the noncancer effects through the inhalation pathway, the risk posed by Ni, Cu, Zn, and Pb was negligible for both adults and children, while Co posed the highest noncancer risk for children. Cobalt also indicated the highest noncancer risk for children through the dermal pathway, while As indicated the highest noncancer risk to children through ingestion. For the cancer risk, the adults were more at risk compared to children, except for As and Co through the dermal pathway posing the highest threat. Trace element concentrations, hazard quotient, and hazard index values indicated that the area was polluted and that noncarcinogenic and carcinogenic effects on residents and miners were significant. Therefore, there is a need to put in place mining regulations aimed at protecting the at-risk human population in the study area.