In this study, a series of fire‐retardant vinyl ester resins (VERs) were obtained via the copolymerization of phosphorus–nitrogen‐containing acrylate [1,3,5‐triglycidyl isocyanurate (TGIC)–acrylic acid (AA)–9,10‐dihydro‐9‐oxa‐10‐phosphaphenanthrene‐10‐oxide (DOPO)] and the widely used vinyl ester resin 901 (901‐VER). The influences of the TGIC–AA–DOPO additive amount on 901‐VER were comprehensively evaluated via the measurement of the flame retardancy, thermal performance, and mechanical properties. In the case of 30 and 40% TGIC–AA–DOPO, the thermosets acquired limited oxygen index values of 30.8 and 31.7%, respectively, and reached vertical burning (UL‐94) ratings of V‐1 and V‐0, respectively. In addition, their peak heat‐release rates, average heat‐release rates, total heat release values, average effective heats of combustion, and total smoke productions decreased significantly. These results demonstrated the great progress of 901‐VER in fireproofing performance. More importantly, the flame‐retardant VER thermosets showed good thermal properties and castings, and glass‐fiber‐reinforced composites based on the developed fire‐retardant VERs possessed comparable tensile strength, flexural strength, and better tensile modulus comparing to those based on neat 901‐VER when the TGIC–AA–DOPO amount was no greater than 30%. From an integrated perspective, 901‐VER with 30% TGIC–AA–DOPO was the best desired. The excellent flame retardancy, thermal performance, and mechanical properties promise the as‐obtained VER important applicable values. © 2019 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2019, 136, 47997.