The development of new polymer compositions characterized by a reduced environmental impact while lowering the price for applications in large-scale production requires the search for solutions based on the reduction in the polymer content in composites’ structure, as well as the use of fillers from sustainable sources. The study aimed to comprehensively evaluate introducing low-cost inorganic fillers, such as copper slag (CS), basalt powder (BP), and expanded vermiculite (VM), into the flame-retarded ammonium polyphosphate polyethylene composition (PE/APP). The addition of fillers (5–20 wt%) increased the stiffness and hardness of PE/APP, both at room and at elevated temperatures, which may increase the applicability range of the flame retardant polyethylene. The deterioration of composites’ tensile strength and impact strength induced by the presence of inorganic fillers compared to the unmodified polymer is described in detail. The addition of BP, CS, and VM with the simultaneous participation of APP with a total share of 40 wt% caused only a 3.1, 4.6, and 3 MPa decrease in the tensile strength compared to the reference value of 23 MPa found for PE. In turn, the cone calorimeter measurements allowed for the observation of a synergistic effect between APP and VM, reducing the peak heat rate release (pHRR) by 60% compared to unmodified PE. Incorporating fillers with a similar thermal stability but differing particle size distribution and shape led to additional information on their effectiveness in changing the properties of polyethylene. Critical examinations of changes in the mechanical and thermomechanical properties related to the structure analysis enabled the definition of the potential application perspectives analyzed in terms of burning behavior in a cone calorimetry test. Adding inorganic fillers derived from waste significantly reduces the flammability of composites with a matrix of thermoplastic polymers while increasing their sustainability and lowering their price without considerably reducing their mechanical properties, which allows for assigning developed materials as a replacement for flame-retarded polyethylene in large-scale non-loaded parts.