Intense ultraviolet irradiation is an important environmental factor affecting the service performance of asphalt mixtures in high-altitude areas, and the asphalt mortar is the main factor affecting the durability of asphalt mixtures. It is of great theoretical significance and engineering value to study the performance of the asphalt mortar at medium and low temperatures under ultraviolet irradiation. Therefore, this paper focuses on the evolution of the effect of the filler content on the rheological properties of different asphalt materials at low and medium temperatures under quantitative UV irradiation. Taking the average amount of UV irradiation observed annually in Northwest China as the indoor aging condition, the matrix asphalt mortar and modified asphalt mortar with different mass ratios of asphalt mortar are selected for indoor aging tests. Physical property tests, low-temperature performance tests, and dynamic shear rheological tests are carried out. The effects of the UV irradiation intensity and mineral powder content on the low temperature performance of the asphalt mortar are studied by variance analysis method, and the reasonable mass ratio range of the asphalt mortar under UV irradiation is proposed based on the standard residual square sum (STRSS) method. The results show that the temperature sensibility and low-temperature deformation energy significantly decrease with the increase in the filler content, while the values of the softening point, fatigue factor (G*sin δ), and creep stiffness modulus of the asphalt mortar increase. In addition, the variance analysis of the creep stiffness modulus aging index (SAI) shows that the ultraviolet radiation intensity has a significant impact on the performance of the asphalt mortar. When the mineral powder content is less than 40%. When the filler content is greater than 40%, the filler content effects the performance of the asphalt mortar. According to the standard residual square sum (STRSS) method, the best mass ratio of the base asphalt mortar is 1.096, and the best mass ratio of the modified asphalt mortar is 0.9091.