Polyethylene terephthalate (PET) drinking bottles, rubber tires, and concrete are the very common municipal solid wastes, which are usually disposed of at landfills and stockpiles and cause continuous damage to the environment. Some studies have indicated that waste PET can be chemically converted into an additive for improving the overall properties of asphalt pavement incorporating natural aggregates, especially the moisture-induced damage resistance. However, it is not clear whether this PET additive still works for asphalt rubber containing recycled concrete aggregates (RCA). To well reveal this issue, this study first adopted a similar way to chemically recycle waste PET into the additive for modifying crumb rubber modified asphalt (CRMA) binder and then mixed the binder with the 13 mm maximum aggregate stone matrix asphalt containing 100% coarse RCA for preparing the mixtures. After a series of physicochemical characterizations of the PET additive, the moisture resistance, rutting resistance, low-temperature cracking resistance, and fatigue resistance of the mixture were systematically evaluated. The results showed that the PET additive is capable of improving the resistance to moisture and high-temperature deformation of asphalt rubber and helps greatly reduce the moisture-induced damage to the interfacial bonding layer. To be more detailed, the residual Marshall stability (RMS) value of RCA-CRMAM/1PET after 72 h of immersion is higher than 85% by contrast to that of RCA-CRMAM (77.1%), while the tensile strength ratio (TSR) value of RCA-CRMAM/1PET shows more than 80% compared to that of 65.2%. In addition, only 1% PET additive can enhance the high-temperature resistance of asphalt rubber containing RCA to rut and allow it to maintain higher resistance to rut after moisture-induced damage. 1% PET additive can help improve the bearing capacity of RCA-CRMAM under a low-temperature environment and delay its fatigue life at small stresses. Generally, with the successful introduction of PET additives to asphalt rubber containing RCA, more durable and sustainable highway pavement can be produced and applied in practice to alleviate the negative impacts caused by waste PET, waste tire rubber, and waste concrete.