Sand has been considered to be something of an immeasurable quantity. There are many indications that this view is no longer valid and that the limiting of natural aggregates usage is doubly justified. Firstly, the extraction of natural aggregates is expensive and has a huge impact on the environment. The main issues in sand and gravel mining are the large areas that are affected, ground water level changes, illegal mining, unsuitability of desert and marine sand, and costs of transport. Secondly, metallurgical waste can be used as a substitute for natural aggregates. This is doubly beneficial—the waste is recycled and the use of natural aggregates is reduced. Waste is stored in landfills that take up large areas and there is also the possibility of ground and groundwater pollution by hazardous compounds. The research presented in this article focuses on the technological conditions of using metallurgical waste in its original form and as a component of recycled concrete aggregate (RCA). The use of metallurgical sludge waste or crushed or round RCA to produce concrete deteriorates the consistency and does not significantly affect the air content and density of the concrete mix. RCA lowers the density of hardened concrete. Metallurgical sludge waste or RCA usage adversely affect the absorbability and permeability of concrete. Concrete containing metallurgical sludge waste is of higher compressive strength after 7 and 28 days, with up to 60% of waste as a sand replacement. RCA concrete achieved higher compressive strength also.