High piezoresistivity of cement-based composites tuned by conductible fillers provides a feasible way to develop self-sensing smart structures and buildings. However, the microstructural mechanisms remain to be properly understood. In the present work, the piezoresistivity of cement mortar with different dosages of graphene nanoplatelets (GNPs) was investigated, and the microstructure was assessed by electron scanning microscopy (SEM) and mercury intrusion porosimetry (MIP). Two surface fractal models were introduced to interpret the MIP data to explore the multi-scale fractal structure of the GNP-modified cement mortars. Results show that the incorporation of GNPs into cement mortar can roughen the fracture surfaces due to the GNPs’ agglomeration. Gauge factor (GF) rises and falls as GNP content increases from 0% to 1% with the optimal piezoresistivity observed at GNP = 0.1% and 0.05%. The GF values of the optimum mortar are over 50 times higher than those of the reference mortar. Fractal dimensions in macro and micro fractal regions change with GNP content. Analysis shows that the fractal dimensions in micro region decrease first and then increase with the increase of GF values. GNPs not only impact the fractal structure of cement mortar, but also alter the tunneling and contact effects that govern the piezoresistivity of composite materials.