In recent years, waste material recycling and reuse have attracted great interest as environmentally friendly modifiers to improve asphalt pavement performance. In this study, anhydrous calcium sulfate whiskers (ACSW), synthesized using phosphogypsum waste, and waste cooking oil (WCO), one of the most prevalent waste oils, were used together as modifiers to create an environmentally friendly asphalt mixture. In particular, WCO was used to compensate for the negative effects of ACSW on asphalt mixture performance at low temperatures. A variety of ACSW and WCO compound-modified asphalt mixtures were fabricated. High-temperature stability, medium-temperature fatigue, low-temperature anti-cracking, moisture susceptibility, repeated freeze–thaw, and long-term aging tests were conducted to comprehensively evaluate the pavement performance. Compared to the base asphalt mixture, the compound-modified asphalt mixtures were demonstrated to have better high- and low-temperature, moisture susceptibility, fatigue, anti-freezing, and anti-aging properties, especially for the 6%ACSW and 2%WCO compound-modified asphalt mixture. Therefore, the 6%ACSW and 2%WCO compound-modified asphalt mixture was ultimately selected for use in construction, as this mixture can meet the requirements for regions with cold winters and hot summers.