In this study, carbon fibers (CFs) were used as toughening materials to improve the mechanical properties of cement stone. The surfaces of the CFs were treated with concentrated nitric acid and sodium hypochlorite to increase the interfacial adhesion between the CFs and the cement. The CFs subjected to surface treatment were evaluated by scanning electron microscopy and infrared analysis to find a significant increase in the number of oxygen-containing groups on the surface. The CFs subjected to surface treatment were added to the cement matrix. The effect of the modified CFs on the mechanical properties of the cement matrix was evaluated by testing the means of mechanical properties. The maximum tensile strength, maximum compressive strength, and ultimate strain of the enhanced cement stone of the CFs treated with sodium hypochlorite increased by 68.2%, 12.0%, and 4.4%, respectively. The maximum tensile strength, maximum compressive strength, and ultimate strain of the enhanced cement stone of the CFs treated with concentrated nitric acid increased by 72.7%, 14.7%, and 4.5%, respectively. The addition of CFs to the cement stone exerted no effect on the type of cement hydration products, as determined by infrared analysis and X-ray diffraction. The toughening mechanism of the modified CFs added to the cement stone was ultimately explored, and the bridging effect, deflection effect, and pull-out effect of cracks were evaluated.