In this work, multi-scale cementitious composites containing short carbon fibers (CFs) and carbon nanofibers (CNFs)/multi-walled carbon nanotubes (MWCNTs) were studied for their tensile stress sensing properties. CF-based composites were prepared by mixing 0.25, 0.5 and 0.75 wt.% CFs (of cement) with water using magnetic stirring and Pluronic F-127 surfactant and adding the mixture to the cement paste. In multi-scale composites, CNFs/MWCNTs (0.1 and 0.15 wt.% of cement) were dispersed in water using Pluronic F-127 and ultrasonication and CFs were then added before mixing with the cement paste. All composites showed a reversible change in the electrical resistivity with tensile loading; the electrical resistivity increased and decreased with the increase and decrease in the tensile load/stress, respectively. Although CF-based composites showed the highest stress sensitivity among all specimens at 0.25% CF content, the fractional change in resistivity (FCR) did not show a linear correlation with the tensile load/stress. On the contrary, multi-scale composites containing CNFs (0.15% CNFs with 0.75% CFs) and MWCNTs (0.1% MWCNTs with 0.5% CFs) showed good stress sensitivity, along with a linear correlation between FCR and tensile load/stress. Stress sensitivities of 6.36 and 11.82%/MPa were obtained for the best CNF and MWCNT-based multi-scale composite sensors, respectively.