Due to periodic variations in temperature and heavy traffic loading, hot-mix asphalt (HMA) pavements undergo considerable distress during their service life. The rheological properties of asphalt binder, when subjected to complex physical and chemical processes, make it stiff and sometimes brittle, which ultimately plays a huge part in pavement deterioration. This phenomenon is commonly known as asphalt aging. Incorporating polymer modifiers with virgin asphalt can work as an effective means to change the binder properties and alleviate the issues related to asphalt aging. Different types of polymers, including elastomers, plastomers, and reactive polymers, can mixed in different combinations with the virgin asphalt to create polymer-modified binders (PMBs). In general, polymers are typically added to the virgin asphalt binder in PMB manufacturing at weight percentages ranging from 3% to 7%. Previous research suggests that many polymer-modified binders (PMBs) show great resiliency and perform extremely well during field and laboratory testing, although the complex nature of asphalt itself makes it significantly difficult to understand the relationship and compatibility of the asphalt–polymer system. This paper aims to develop a comprehensive literature review on the chemical aspects, microscopic structure, and compatibility of polymers with virgin asphalt. It was found that swelling, storage stability, blend morphology, and the polymer mixing technique play a great role in the compatibility of asphalt–polymer systems. Thermoplastic elastomers (e.g., styrene–butadiene–styrene) and plastomers (e.g., ethylene–vinyl acetate) are the most used polymer modifiers for asphalt binders. The compatibility of the polymer–asphalt system can be improved by sulfur vulcanization, antioxidants, hydrophobic clay minerals, functionalization, and reactive polymers, among other techniques.