The sol-gel process has been applied to cotton fabrics to cover the fibers with a silica-based film, which can improve their thermal oxidation and combustion behaviors. Silica sol, silane coupling agent 3-glycidoxypropyltrimethoxysilane (KH560) and flame retardant zinc borate (ZB) are used together to prepare SiO2-KH560-ZB hybrid sol, which is then finished on the surface of cotton fabric through impregnation and baking. Fourier transform infrared spectroscopy (FTIR), thermogravimetric analysis (TGA), limiting oxygen index analysis (LOI) and microscale combustion calorimetry (MCC) are used to characterize functional groups, thermal stability and flammability properties of finished fabrics. Surface morphology of sol modified cotton fabrics are analyzed by scanning electron microscopy (SEM) and energy dispersive X-Ray spectroscopy (EDX). Results show the sols are successfully converted into gel coatings onto cotton fabrics, and continuous smoldering phenomenon of resulted fabrics disappears. ΔLOI/Δm of SiO2-KH560-ZB coated cotton fabric reaches the highest of 0.37%/g, char residue rate is as high as 28.43% and the peak heat release rate (PHRR) is reduced by 26.9% (83.7 W/g). KH560 has a significant coupling effect on combining components in the sol system and increases compatibility between sols and cotton fabrics. ZB (cooling, separation, dilution and suppression), silica sol (physical barrier) and KH560 show excellent synergistic effects in enhancing overall flame retardancy.