A trade‐off dilemma exists for simultaneously improving the mechanical properties and flame resistance of flexible polyvinyl chloride (fPVC)/magnesium hydroxide (MH) composites. In this study, epoxy phosphate ester (EPE), a hydrophobic surface modifier of MH, was synthesized using a self‐emulsification method. After modification, EPE was bonded to the surface of MH (MHEPE) without altering its morphology. The results of limiting oxygen index and cone calorimetry tests indicated that fPVC/MHEPE exhibited better flame retardancy and smoke suppression effects than did fPVC/MH. The peak of the heat release rate, total heat release, peak of the smoke production rate, and total smoke production of the fPVC/MHEPE composite were 206.0 kJ m−2, 45.90 MJ m−2, 0.0729 m2 s−1, and 9.88 m2, which were 8.64%, 14.00%, 27.61%, and 9.02% lower than those of the fPVC/MH composite, respectively. For the fPVC/MHEPE composite, a compact and continuous char residue formed, which could inhibit heat and flammable volatile migration between the matrix and burning zones. In the gas phase, the dilution effect of H2O vapor reduced the concentrations of O2 and flammable volatiles. The free‐radical quenching effect of ·PO and ·PO2 also played a vital role in extinguishing flame and terminating combustion. Further, the introduction of EPE improved the tensile and impact strengths of the fPVC/MH composites because of the excellent interfacial compatibility between MHEPE and the fPVC matrix. This study provides a simple and workable solution for the trade‐off dilemma, and the remarkable flame retardancy and mechanical properties of the fPVC/MHEPE composite render it a promising cable material.