A system based on a nitrogen–phosphorous flame retardant, melamine polyphosphate (MPP), and aluminum phosphinate (OP), was employed to flame‐retard glass fabric‐reinforced epoxy resin composites (GFEP). Surface modification was adopted to effectively solve the aggregation of MPP and OP particles in EP matrix during the curing process. The flammability characteristics and synergistic effect have been studied by using limiting oxygen index (LOI), vertical burning (UL94) tests, thermogravimetric analysis, and pyrolysis‐gas chromatography/mass spectrometry. The flame resistance tests indicated that the synergism system with an optimized ratio achieved UL94 V‐0 (1.6 mm) rating and the LOI value of 33.5%. The chemical composition and structure of residue were characterized by X‐ray photoelectron spectroscopy, Fourier transform infrared, and scanning electron microscopy. It is concluded that the synergistic system having a balance of nitrogen element and phosphorous element accelerate the carbonation of EP resin, which was in favor of the construction of compact and continuous charring structure, thus obtaining satisfactory flame retardance. POLYM. COMPOS., 40:3199–3208, 2019. © 2018 Society of Plastics Engineers