This paper aims to optimize the asphalt-mortar interface characteristics of semi-flexible pavement coating material (SFPCM), three types of the interface modifiers, namely the silane coupling agent, carboxylated styrene-butadiene latex, and cationic emulsified asphalt, are selected to add into the cement mortar to optimize the asphalt-mortar interface. The microscopic characteristics of the asphalt-mortar interface and the macroscopic pavement performance of the SFPCM after the interface optimization are investigated to reveal the effect of interface modifiers and determine the best formula. Results show that: (1) The dry shrinkage of the cement mortar is increased with the increased dosage of the interface modifiers, while the strength is decreased accordingly; (2) the optimum dosages of the silane coupling agent, carboxylated styrene butadiene latex, and cationic emulsified asphalt are 0.5%, 10%, and 5%, respectively; (3) the microscopic asphalt-mortar interface characteristics of the SFPCM with different interface modifiers have significant differences; (4) the SFPCM with the cationic emulsified asphalt presents the best pavement performance, particularly the high-temperature stability. These results can be attributed to improve the pavement properties of SFPCM.