In this study, basalt fiber having two types of diameters (16 μm and 25 μm) was selected and added to asphalt mastic and asphalt mixtures using different fiber proportions. The influences of fiber diameters and proportions on the properties of asphalt mastic and mixtures were studied. The adhesion behavior of the fiber-asphalt mastic (FAM) interface was evaluated by a monofilament pullout test, and the rheological properties of FAM were evaluated by temperature sweep, linear amplitude sweep, and bending beam rheological tests. In addition, the high-temperature stability, intermediate and low-temperature cracking resistance, and water stability of fiber-modified mixtures were studied by wheel tracking, ideal cracking, a low-temperature bending beam, and a water-immersed Marshall test. The results showed that the interface adhesion behavior between 16 μm fiber and asphalt mastic was more likely in the fiber failure mode at both −12 °C and 25 °C. Adding basalt fiber can significantly improve the high-temperature and fatigue properties of asphalt mastics. Moreover, 16 μm fiber had a better modifying effect on asphalt mastic than 25 μm fiber. The same enhancement trend can be observed in asphalt mixtures. Basalt fibers with 16 μm diameters can improve the high-temperature performance of asphalt mixtures more significantly. In addition, 16 μm fiber could sharply enhance the cracking performance of the mixtures at intermediate and low temperatures, while the enhancing effect of 25 μm fiber on the mixture is insignificant, though both diameters of the fibers have a minor effect on the water stability.