The polystyrene (PS) composite containing self‐expanded intumescent flame retardant (polyphosphate ammonium and expandable graphite) was blended with three butyltriphenylphosphine‐based chelate borates, respectively, to evaluate their effect on flame retardancy. The chemical structure of as‐prepared three chelate borates was confirmed by nuclear magnetic resonance (NMR) and Fourier transform infrared spectrum (FTIR). The flame retardancy of various PS composites was evaluated by vertical burning test (UL‐94), limited oxygen index (LOI), and cone calorimeter (CC). Flammability and combustion results suggested that one of chelate borates, named [BTP][BMB], made PS composite (PS4) obtain V‐0 rating, 27.0 ± 0.3% LOI value, and reduction on heat release and smoke production with 17 wt% total flame retardants loading. The combustion residue was analyzed by scanning electron microscope and FTIR, and the pyrolysis gaseous products were investigated by TG‐FTIR technique. Besides, complex viscosity of PS composites composed of various chelate borates from a rheology instrument indicated that the improvements of flame retardancy of PS composites depended on the temperature of construction of crosslinked network by expandable graphite, which the chelate borates showed distinctive influence. Accordingly, the flame‐retarding mechanism about fast response to flame has been proposed.