To enhance the flame retardancy of 9,10‐dihydro‐9‐oxa‐10‐phosphaphenanthrene‐10‐oxide (DOPO) on epoxy resins (EPs), by addition reaction with allyltriphenylphosphonium bromide (ATPB), a novel DOPO‐containing phosphonium bromide (DOPO‐ATPB) was synthesized and its structure was characterized by Fourier transform infrared (FTIR), 13C nuclear magnetic resonance (NMR), and X‐ray photoelectron spectroscopy (XPS). By intercalating DOPO‐ATPB into montmorillonite (MMT) layers and with the help of solvents, the intercalated nanocompound (DPAB‐MMT) was prepared, and X‐ray diffraction (XRD) confirmed a d‐spacing expansion of DPAB‐MMT layers (from ~1.01 to ~2.10 nm). The epoxy composites with DOPO‐ATPB and DPAB‐MMT were prepared, and their effects on the flame retardancy, mechanical and dielectric properties were studied by limiting oxygen index (LOI), UL‐94 vertical burning test, cone calorimetry, three‐point bend testing, and the impedance analyzer. The results show that DOPO‐ATPB endows EPs with excellent flame retardancy and dielectric properties, and DPAB‐MMT can be well dispersed in EPs. Compared with pure EP, the EP/8 wt% DOPO‐ATPB increased the LOI values from 23.0% to 32.5%, achieved V‐0 rating and decreased the peak heat release rate (p‐HRR) by approximately 36.6%. With the 8 wt% DPAB‐MMT, the LOI values of epoxy composite was increased from 23.0% to 35.9%, the p‐HRR was decreased by ~25.4%, the mechanical flexural strength was enhanced by approximately 13.4% and the dielectric loss (ε″) was decreased from 0.080 to 0.052 at 100 Hz.