Asphalt pavement construction is a large-volume project, with the ability to recycle the industrial waste and reduce carbon emissions. Rubber-modified asphalt is a carbon-neutralized asphalt-based material, facilitating the recycling of waste rubber materials and improving the road performance of the asphalt mixture. To evaluate the interface interaction of the rubber–asphalt system and its effect on the viscosity characteristics of rubber-modified asphalt, the contact properties of rubber particles in asphalt were analyzed on a microscopic level. Rubber swelling tests and solvent elution tests were conducted on the rubber–asphalt system under different preparation conditions. The swelling ratio, degradation ratio, and swelling–degradation ratio were proposed to evaluate the interface interaction. The results show that the interface interaction of the rubber–asphalt system can be divided into the following three stages: swelling, effective degradation, and over-degradation. The degree of swelling is mainly affected by the content and size of the rubber particles and it is physically condensed, while the degradation is mainly affected by the preparation temperature and preparation time. The effective interface interaction greatly affects the viscosity with the building of the stable three-dimensional network structure. The stronger the interface interaction, the greater the viscosity of the rubber-modified asphalt, except for the 25% content of rubber particles. The gel film will be generated on the surface of the rubber particles throughout the swelling and effective degradation, increasing the viscosity of the rubber-modified asphalt.