Construction and demolition (C&D) materials obtained from the demolition of buildings are proven to be qualified and sustainable subgrade fillers. The permanent deformation response of subgrade C&D materials under different moisture contents, degrees of compaction, deviator stresses, and confining pressures was revealed by carrying out dynamic triaxial texts. Then, using a four-factor and three-level orthogonal test and by calculating the Gray correlation degree of each factor, the influence degree of each factor on the permanent deformation was determined. The results indicated that two different response types of the permanent deformation of subgrade C&D materials, plastic shakedown and plastic creep, were identified as reason behind the increase in stress levels. Also, according to the Gray correlation analysis results, the permanent deformation of highway subgrade filled with C&D materials is influenced by the deviator stress most significantly, followed by moisture content, degree of compaction, and confining pressure. Finally, a permanent deformation prediction model about this kind of subgrade filler with a reasonable prediction accuracy was proposed.