The evolution of frictional strength during stick-slip dynamics of a fault system is key to understanding the earthquake nucleation and rupture patterns. In mature faults, granular gouge is produced by wear, comminution or fragmentation during tectonic movements. In this work, we introduce a fragmentation model in the simulation of a sheared granular fault to explore the influence of grain breakage on the stick-slip dynamics. With fragmentation of highly stressed particles, the fault frictional strength increases accompanied with many small slip events triggered by particle breakage. The small fragments produced by particle breakage are not only stronger and more difficult to break, but they also change the distribution of contact forces, leading to strengthening of the fault system. By statistical analyses on size distribution of slip events under different particle strengths, we find that when particles are weaker, slip events become more correlated with particle fragmentation events and that the number of large slip events decreases. In addition, our analyses on the relationship between slip and particle fragmentation events reveal three types of correlations: in the first and second types, particle fragmentation events trigger micro-or major slips, respectively. In the third category, large-scale particle fragmentations take place at the end of large slip events owing to stress localization during post-slip particle rearrangements. Our results in this work highlight the role of micromechanics of particle fragmentation in failure of fault damage zones and help in understanding the relation between particle breakage and frictional failures.