Plastic production has risen steadily, but recycling rates lag. Researchers are increasingly investigating the use of plastics in road construction, especially in terms of modifying asphalt with waste plastics. The dry process, which involves incorporating plastics into hot aggregates, is increasingly gaining traction as an alternative to the wet process, where plastics are added to hot bitumen. Past studies indicate enhanced asphalt mixture properties with the dry process, but there is debate about the role of waste plastics—whether they should be used as aggregates, fillers, or binder modifiers. This study explores the extent to which dry-added waste plastic modified the binder of the asphalt mixtures. Fluorescent microscopy and scanning electron microscopy revealed the impact of plastic on the binder, while image analysis quantified polymer swelling and dispersion in the binder matrix. It was concluded that when plastics are added to hot aggregates, they will act as binder modifiers. Lower plastic content and reduced polymer crystallinity led to increased polymer swelling and better dispersion in the mixture. This study recommends plastic inclusion of less than 2.5% (by volume) in the dry-added method since high plastic content leads to polymer agglomeration, especially for highly crystalline polymers. Additionally, mixes modified with amorphous plastics exhibited superior workability and performance compared to those modified with crystalline plastics. This study also suggests that using plastics to replace both bitumen and filler can improve cost efficiency, reduce the carbon footprint, and enhance the overall performance of the asphalt mixture.