The bisphenol F epoxy resin (DGEBF) reacted with 10-(2,5-Dihydroxyphenyl)-10H-9-oxa-10-phospha-phenantbrene-10-oxide (ODOPB) and phenyltrimethoxysilane (PTMS) to obtain a novel epoxy resin containing both phosphorus and silicon (EP-P/Si). EP-P/Si exhibited evidently improved flame retardancy, with a limited oxygen index value of 33.4% and UL-94 V-1 rating acquired. In cone calorimeter test, its peak heat release rate (PHRR), total heat release (THR), average effective heat of combustion (av-EHC), and total smoke production (TSP) were reduced by 36.0%, 19.5%,11.5%, and 7.2% compared with neat epoxy resin (EP), respectively, indicating that the P/Si synergistic effect not only improved the flame retardancy but also inhibited the smoke release. The flame retardancy mechanism was studied by analysis of char residue and pyrolysis behavior in gas phase. Scanning electron microscopy (SEM) results exhibited that EP-P/Si formed a dense and compact carbon layer acting as a barrier to inhibit further combustion. And the Fourier transform infrared (FTIR) spectra, laser Raman spectroscopy (LRS), and X-ray photoelectron spectroscopy (XPS) results indicated that it had good thermal stability. In addition, the pyrolysis-gas chromatography/mass spectrometry (Py-GC/MS) results suggested that the phosphorus-containing radicals (·PO2) that had quenching effect existed in the gas phase. While the flame retardancy got improved, EP-P/Si also exhibited excellent mechanical properties, with an improvement of 31.8%, 6.2%, and 369.7% in tensile strength, flexural strength, and impact strength compared with EP, respectively.