In this paper, an organic/inorganic phosphorus and silicon-containing coating was prepared by the sol–gel method and its flame retardant mechanism for poly(ethylene terephthalate) (PET) fabrics was thoroughly investigated. The influence of hybrid coating on PET fabric was investigated by the limited oxygen index (LOI), vertical flammability test (VFT) and microscale combustion calorimetry (MCC). The results revealed that the LOI of the coated PET fabric was up to 29.2% and no dripping was observed in the VFT. MCC results showed that hybrid coating reduced the peak heat release rate, heat release capacity and total heat release of PET fabric. Thermogravimetric analysis demonstrated increased thermal stability and char residue of PET fabrics due to the hybrid coating. Furthermore, the char residues were studied by X-ray photoelectron spectroscopy and scanning electron microscopy. The above results proved that the synergistic effect between the 9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide group and the silicon network played a key role in the flame retardancy of PET fabrics. In addition, the durability of coated PET fabrics was interpreted by calculating the interaction energy through molecular dynamics simulation.