CO2 laser polishing process can significantly improve the surface quality and the laser-induced damage threshold (LIDT) of fused silica optics. However, due to the thermal history of the laser polishing process, the increment of the fictive temperature inside the modification layer would cause densification and residual stress, which critically affect the surface accuracy and the service life of fused silica optics. In this work, a 3D multi-physical coupling model including temperature, fluid flow and fictive temperature was established. Based on the fictive temperature distribution of the fused silica polished by CO2 lasers, the mechanism of laser annealing on the modified layer was revealed. The annealing results of fused silica were defined as three states including incomplete annealing, perfect annealing and over annealing. Based on the simulation results, the fictive temperature inside the modified layer was completely reduced with no increment of modified layer depth under the perfect annealing state. Additionally, the residual stress and the fictive temperature after the laser annealing were characterized by the Raman spectrum. The fictive temperature was reduced by 16.8 % and the residual stress was effectively reduced. This work can provide theoretical and experimental guidance for the control of surface modification and residual stress of fused silica optics polished by CO2 lasers.