To solve the problem of the poor abrasion resistance of concrete pavement surface mortar, this study substituted cement with equal amounts of styrene–butadiene rubber (SBR) latex and silica fume (SF) to investigate the effects of organic/inorganic material composite modification on the fluidity, drying shrinkage, mechanical properties, and abrasion resistance of cement mortar. Also in this study, the microstructure, product, and pore structure characteristics of the composite modified cement mortar were investigated using scanning electron microscope (SEM), X-Ray diffraction (XRD), Fourier-transform infrared spectroscopy (FT-IR), and the Brunauer–Emmett–Teller (BET) method. This research found that the sole substitution of SF negatively impacted the mortar’s fluidity and drying shrinkage yet enhanced its mechanical strength and abrasion resistance; the incorporation of SBR latex improved fluidity, reduced shrinkage, and increased flexural strength but adversely affected the compressive strength of the mortar. Additionally, the enhancement of the mortar’s abrasion resistance with SBR latex was significantly greater than that with SF. When SBR latex and SF were used together as substitutes, the latex struggled to offset the negative impact of SF on mortar fluidity but effectively reduced shrinkage; SF compensated for the detrimental effect of the latex on compressive strength. Moreover, the primary role in enhancing the mortar’s abrasion resistance was played by the latex. Microscopic tests showed that SBR latex and SF could increase the content of calcium silicate hydrate (C-S-H) gel, inhibit the formation of ettringite (AFt) and reduce carbonation, refine the pore size of cement mortar, and effectively improve the microstructure of mortar.