The level of radionuclides is an important index for the preparation of building materials from industrial solid waste. In order to investigate the radiological hazard of five kinds of typical general industrial solid wastes in Guizhou, China, including fly ash (FA), red mud (RM), phosphorus slag (PS), phosphogypsum (PG), and electrolytic manganese residue (EMR), the radiation intensity and associated radiological impact were studied. The results show that concentrations of 238U, 235U, 232Th, 226Ra, 210Pb, and 40K for different samples vary widely. The concentration of 238U was both positively correlated with 235U and 226Ra, and the uranium contents in the measured samples were all of natural origin. The radiation levels of PG, EMR, EMR-Na (EMR activated by NaOH), and EMR-Ca (EMR activated by Ca(OH)2) were all lower than the Chinese and the world’s recommended highest levels for materials allowed to be directly used as building materials. The values of the internal and external illumination index (IRa and Iγ, respectively) for FA and RM were higher (IRa > 1.0 and Iγ > 1.3 for FA, IRa > 2.0 and Iγ > 2.0 for RM). The radium equivalent activity (Raeq), indoor and outdoor absorbed dose (Din and Dout, respectively), and corresponding annual effective dose rate (Ein and Eout) of RM, PS, and FA were higher than the recommended limit values (i.e., 370 Bq/kg, 84 nGy/h, 59 nGy/h, 0.4 mSv/y, and 0.07 mSv/y, respectively), resulting from the higher relative contribution of 226Ra and 232Th. The portion of RM, FA, and PS in building materials should be less than 75.44%, 29.72%, and 66.01%, respectively. This study provides quantitative analysis for the safe utilization of FA, RM, PS, PG, and EMR in Guizhou building materials.