Two PEPA-based (PEPABFR) and DOPO-based (DOPOBFR) flame retardants with vinyl groups were synthesized successfully via 1-oxyphospha-4-hydroxymethyl-2,6,7-trioxabicyclo[2.2.2]octane (PEPA) or 9,10-dihydro-9-oxa-10-phosphaphenanthrene 10-oxide (DOPO) with isocyanoethyl methacrylate using a one-pot method. Their structures were confirmed using Fourier transform infrared, 1 H NMR and 31 P NMR spectroscopies and matrix-assisted laser desorption ionization time-of-flight mass spectrometry. The mechanical performance, thermal properties and flame retardant performance of systems with 15 wt% synthetic flame retardants were investigated using bending tests, differential scanning calorimetry, thermogravimetric analysis, limiting oxygen index (LOI) measurements and cone calorimeter tests. The results show that the bending strength and flexural modulus are improved significantly. The LOI increases to 29.3% and 30.7% with incorporation to 15 wt% PEPABFR and 15% wt % DOPOBFR from the 20.3% corresponding to pure vinyl ester resin (VER). The peak heat release rate decreases by 16.9% and 39.3% and total heat release decreases by 20.1% and 36.4% for VER/PEPABFR and VER/DOPOBFR systems. Furthermore, the flame retardancy mechanism of the flame retardants was studied using thermogravimetric analysis coupled with Fourier transform infrared spectroscopy, scanning electron microscopy and X-ray photoelectron spectroscopy.