DOPO‐based dicarboxylic acid (DODA) was synthesized and used to prepare DOPO‐based polyamide (DOPA) using a direct polycondensation reaction. Also, DODA was used for the surface modification of Mg(OH)2 nanoparticles. Polyamide/Mg(OH)2 nanocomposites (DPANC) were prepared based on DOPA and the organically modified Mg(OH)2 nanoparticles through the solvent casting method. Further, to investigate the effect of the DOPO structure on the thermal and combustion properties, a poly(amide‐imine) (PAIm) without the DOPO structure was synthesized. The structure and morphology of the prepared nanocomposites were studied using various techniques. The results indicated that nanoparticles were uniformly dispersed in the polymer matrix. According to thermogravimetric analysis results, the temperature at 10% mass loss was increased from 309 to 337 °C in the DPANC sample containing 8 mass% of nanoparticles as compared to DOPA in inert atmosphere. Further, according to microscale combustion calorimetry results, the peak heat release rate (pHRR) was decreased from 33 to 19 W g−1 in the DPANC 5 sample, as compared to DOPA. Also, the pHRR value of PAIm was obtained as 138 W g−1, which was about 105 W g−1 higher than that of polyamide. © 2020 Society of Industrial Chemistry