With the gradual depletion of natural sand due to over-exploitation, alternative building materials, such as manufactured sand aggregate (MSA), have attracted much attention. In order to interpret the evolution of pore structure and fractal characteristics in MSA mortar over long-term water saturation, the 1H low-field nuclear magnetic resonance (LF-NMR) relaxation method was used to investigate the temporal evolution of the pore structure in five single-graded MSA mortars and synthetic-graded mortars with small amplitudes in particle size. MSA presents a fresh rock interface characterized by a scarcity of pores, which significantly reduces the porosity of the mortar. The surface-to-volume ratio (SVR) is employed for characterizing the MSA gradation. Through an analysis of parameters, such as total porosity, pore gradation, pore connectivity, and pore fractal dimension of mortar, a correlation model between pore structure parameters and aggregate SVR is constructed. The fractal characteristics of pores and their variations are discussed under three kinds of pore gradations, and the correlation model between fractal dimension and porosity is established. These results demonstrate the high impermeability and outstanding corrosion resistance of synthetic-graded mortar. The fractal model of the pore structure evolution of MSA mortar has a guiding effect on the pore distribution evolution and engineering permeability evaluation of MSA mortar in engineering.