Ablation‐resistant polymeric composites play a crucial role in the thermal protection system for the entire aerospace industry. Constructing a stable char layer especially with low thermal conductivity is essential to enhance the ablation resistant properties of polymer‐based ablatives. In this article, basic magnesium carbonate (MgCO3) and carbon fibers (CFs) were introduced to silicone rubber with an aim of building fiber‐supported micro‐porous char structure in situ, thereby improving ablation resistance properties. Thermogravimetric analyses revealed that the addition of MgCO3 significantly increased the char residue of the composites. Oxyacetylene torch tests indicated that the addition of MgCO3/CFs greatly improved the thermal insulation of the composites, and the linear ablation rates of the composites were reduced by 30.76% compared to the virgin silicone rubber. Shore A hardness and thermal conductivity tests revealed that composites with 10 phr MgCO3 exhibited a char structure with proper strength and low thermal conductivity. SiC, Mg2SiO4, and MgSiO3 were produced in the MgCO3/CFs‐modified composites, as characterized by X‐ray diffraction. Combined with macro lens and scanning electron misroscopy, it was proven that a fiber‐supported micro‐porous and thermally insulated char structure was conducive to the improvement of the ablation properties, which can be applied to aerospace systems for thermal shielding applications.