This study aims to investigate the effect of calcium sulfate whisker (CSW) on the properties and microstructure of cement-based composites. Further, nanosilica (NS) was used as a comparison. The results show that the compressive strength and fracture toughness of cement-based composites increased by 10.3% and 10.2%, respectively, with 2% CSW. The flexural strength, splitting tensile strength, and fracture energy increased by 79.7%, 34.8%, and 28.7%, respectively, with 1% CSW. With the addition of CSW, shrinkage deformation was aggravated, and the capillary water absorption coefficients were clearly reduced. Compared with NS, CSW-reinforced cement-based composites show better tensile, flexural, and fracture properties and smaller shrinkage deformations and capillary water absorption coefficients. The residual mechanical properties of all groups improve when the treating temperature is lower than 400 °C and decline rapidly when the temperature goes over 600 °C. When treated at 900 °C, the residual mechanical properties are 40% less than those at ambient temperature, with the NS group showing the best performance, followed by the control group and the CSW group. X-ray diffraction (XRD) and scanning electron microscopy (SEM) tests show that the addition of CSW improves the microstructure of the matrix. CSW can reinforce and toughen composites by generating ettringite and whisker pullout, whisker breakage, crack bridging, and crack deflection at the microstructure level.