The construction of sponge city is a major green innovation to implement the concept of sustainable development. In this study, the road performance of permeable asphalt concrete (PAC), which displays pronounced water permeability and noise reduction that are favorable for sponge cities, has been improved with a two-fold modification using styrene–butadiene–styrene (SBS) and crumb rubber (CR). Four percent SBS and three different ratios (10%, 15%, and 20%) of CR have been used to modify the virgin asphalt binder. The Marshall design has been followed to produce PAC samples. To evaluate the asphalt binder performance, multiple-stress creep-recovery (MSCR) test, linear amplitude sweep (LAS) test, and engineering property test programs including softening point test, penetration test, and rotational viscosity test have been conducted. Freeze–thaw splitting test, Hamburg wheel-tracking test, resilient modulus test, and permeability coefficient test have been performed to evaluate the asphalt mixture performance. The test results show that the addition of SBS and CR reduces the permeability coefficient, but significantly improves the high temperature performance, fatigue performance, and rutting resistance as well as the resilient modulus. However, the optimum rubber content should not exceed 15%. Meanwhile, after adding CR and SBS modifier, the indirect tensile strength (ITS) and tensile strength ratio (TSR) increase. It indicates that the moisture stability and crack resistance have been improved by the composite modification effect.