In recent years, diatomite has been successfully adopted in asphalt modification to overcome the problems of polymer modified asphalt, because of the advantages in wide sources, low price, and worthy technical characteristics. Although the improvement of the high-temperature performance of the modified diatomite asphalt has been verified in previous studies, the diatomite will bring negative impact on the low-temperature resistance. Hence, the objective of this study is to seek a new channel to improve the comprehensive performance of the diatomite modified asphalt binder. Considering the advantage of the SBR in improving the low-temperature performance of asphalt binder, the diatomite/SBR composite modified asphalt binder (DSA) and the corresponding preparation technology are developed to obtain an improved comprehensive performance via the orthogonal experiment method in this study. Moreover, the modification mechanisms of the DSA are revealed using fluorescence microscopy (FM) tests, Gel permeation chromatography (GPC) tests, and Fourier transform infrared spectroscopy (FTIR) tests.