In this study, thermoplastic poly(ester ether) elastomer (TPEE) nanocomposites with phosphorus–nitrogen (P–N) flame retardants and montmorillonite (MMT) were prepared by melt blending. The fire resistance of the nanocomposites was analyzed by limiting oxygen index (LOI) and vertical burning (UL 94) tests. The results show that the addition of the P–N flame retardants increased the LOI of the material from 17.3 to 27%. However, TPEE containing P–N flame retardants only obtained a UL 94 V‐2 ranking; this resulted in a flame dripping phenomenon. On the other hand, TPEE containing the P–N flame retardant and organically modified montmorillonite (o‐MMT) achieved better thermal stability and good flame retardancy; this was ascribed to its partially intercalated structure. The synergistic effect and synergism were investigated by Fourier transform infrared spectroscopy and thermogravimetry. The introduction of o‐MMT decreased the inhibition action of the P–N flame retardant and increased the amount of residues. The catalytic decomposition effect of MMT and the barrier effect of the layer silicates are discussed in this article. The residues after heating in the muffle furnace were analyzed by scanning electron microscopy, energy‐dispersive X‐ray spectroscopy and laser Raman spectroscopy. It was shown that the intercalated layer silicate structure facilitated the crosslinking interaction and promoted the formation of additional carbonaceous char residues in the formation of the compact, dense, folded‐structure surface char. The combination of the P–N flame retardant and o‐MMT in TPEE resulted in a better thermal stability and fire resistance because of the synergistic effect of the mixture. © 2014 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2014, 131, 41094.