To promote the effective utilization of sludge and slag produced in nature and from human activities, this paper summarizes the research progress in the field of building materials on the basis of expounding their classification and characteristics. (1) Sludge and slag include silt, sludge and industrial waste residues. These three materials are mainly composed of SiO2, which can be used to produce building materials after treatment and can also be used as admixtures, including roadbed admixtures. (2) Silt and sludge are widely used in building wall materials and roadbed materials, etc. Industrial waste residues can be used in the production and processing of green concrete and glass-ceramics. (3) In addition to continuing to use existing utilization methods, key treatment technologies and new treatment devices can be further developed according to the characteristics of sludge and slag. Moreover, observations and mechanistic analysis of the microscopic structure of industrial waste residues and research on strong and weak utilization methods based on the performance of building materials can be carried out, and more efficient and energy-saving excitation or activation technologies will be developed. These efforts will eventually lead to the development of functional building materials with excellent performance and environmentally friendly characteristics to achieve the differentiated utilization of silt, sludge, and industrial waste residues and realize the efficient transformation of resources. This paper provides useful insights for the application of sludge and slag in the field of building materials.