Silicones are often used for various types of coatings, but due to their poor mechanical properties, they often require modification to meet specific requirements. At the same time, various production processes throughout the world generate different types of waste, the disposal of which is harmful to the environment. One possible solution is to use production waste as a filler. In this paper, the authors investigated how the use of metallurgical production waste products as fillers changed the mechanical properties of silicone composites prepared by casting. Composite samples were characterized using tensile tests, resilience, pin-on-disc, Schopper–Schlobach abrasion, hardness, and density measurements. Based on the obtained results, the authors assessed the effect of each of the fillers used in different weight proportions. The results showed that the silicone composite filled with 5 wt% zinc dust showed the lowest decrease in tensile strength and Young’s modulus, with a simultaneous significant reduction in abrasion compared with the reference sample. This research shows that zinc waste can be successfully introduced into a silicone matrix in cases where it is important to reduce abrasive wear.