The aluminum recycling industry produces aluminum filter dust (AFD), a waste byproduct of the aluminum recycling process composed mainly of aluminum oxide in a percentage between 60–70%, 8% calcium oxide, almost 15% sodium chloride, and between 5–10% potassium chloride. Due to its aluminum content, this waste can be used as a raw material in the manufacture of ceramic bricks, at the same time reducing the environmental impact produced in landfill. In this work, the partial substitution of a clay mixture (40% black, 30% red, and 30% yellow clay) by different proportions of AFD in the range 0–25 wt % for the production of fired clay brick was studied. The raw materials, clays, and waste were characterized by XRF and XRD. The brick specimens were fired at 950 °C and their physical and mechanical properties, such as water absorption, water suction, loss of ignition, linear shrinkage, bulk density, and compressive strength, were analyzed. The more relevant results were obtained with the addition of up to 20 wt % AFD, obtaining bricks with physical properties comparable to pure clay-based bricks used as a reference and better compressive strength and thermal conductivity due to the balance between the melting and pore-forming effects of the waste. These sustainable bricks also comply with the regulations of heavy metals leached to the environment, as indicated by the leaching test.