The influence of the specific surface area of aluminum trihydrate (ATH) on the synergism to different phosphorus‐based flame retardants (FRs) in unsaturated polyester (UP) resin was investigated. Flammability and thermal stability of UP composites were evaluated with UL‐94 vertical burning, thermogravimetric analysis, and cone calorimetry. The morphology of char residues was examined via scanning electron microscopy (SEM). An interaction between crystalline ATH (4.5 m2 g−1) and aluminum hypophosphite (AHP) or zinc diethylphosphinate (DEPZn) was observed. When 37 wt % ATH and 8 wt % AHP or DEPZn were incorporated, samples passed V‐0 in UL‐94 tests. Furthermore, peak of heat release rate (PHRR) and total heat release (THR) were reduced by introduction of phosphorus containing FRs compared to a formulation containing solely crystalline ATH. The incorporation of amorphous ATH (300 m2 g−1) leaded to lower UL‐94 ratings, using the same amount of AHP or DEPZn. In comparison to crystalline ATH, an addition of phosphorus containing FRs did not lead to a decrease of PHRR and THR for amorphous ATH. SEM of the residues showed that the combination of crystalline ATH with phosphorus species having hydrogen/carbon rich environment formed a dense and tough layer. The combination of crystalline ATH with phosphorus species providing oxygen rich environment gives fragile residues. An incorporation of amorphous ATH leaded to a fragile residue as well. © 2018 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2019, 136, 47270.