To understand the effects of filler-asphalt ratio on different properties of lignin and polyester fiber reinforced shape memory polyurethane (SMPU)/styrene butadiene styrene (SBS) composite modified asphalt mortar (PSAM), as well as to reveal the reinforcing and toughening mechanisms of lignin and polyester fibers on PSAM, SMPU, SBS and mineral powder were first utilized to prepare PSAM. Then the conventional, rheological and anticracking properties of lignin fiber reinforced PSAM (LFAM) and polyester fiber reinforced PSAM (PFAM) at different filler-asphalt ratios were characterized. Test results indicate that the shear strength, deformation resistance and viscosity are increased after adding 0.8wt% lignin fiber or polyester fiber and increasing the filler-asphalt ratio from 0.8 to 1.2. The optimal filler-asphalt ratio of 1.0 is proposed after comprehensive performance assessments of PSAM. Polyester fiber shows a better reinforcing effect than lignin fiber, but its improvement in the thermal stability of PSAM is not significant at high temperatures. Additionally, the complex modulus, storage modulus, loss modulus and rutting resistance factor of PSAM are improved after adding lignin fiber and polyester fiber, as well as show an increasing trend as the filler-asphalt ratio is raised, but the phase angle is gradually decreased. Further, the increase of elastic components in PSAM effectively enhances the anti-deformation ability of PSAM at high temperatures, and polyester fiber more obviously improves the high-temperature deformation resistance of PSAM than lignin fiber. Finally, the anti-cracking performance of PFAM and LFAM at low temperatures is reduced by 74.2% and 46.7%, respectively, as the filler-asphalt ratio is raised from 0.8 to 1.2. The low-temperature anti-cracking performance of LFAM is lower than that of PFAM at the same filler-asphalt ratio, even lower than that of PSA. Compared with lignin fiber, the anti-cracking performance and deformation resistance of PSAM at low temperature is more greatly enhanced by polyester fiber.