Because of low resilient modulus, high plasticity soil is often not allowed to fill road subgrades and is discarded as construction and demolition waste (CDW). To make use of the CDW, this study explored the possibility of improving high plasticity soil with gravel and examined the effect of gravel gradation on the resilient modulus of the soil-gravel mixture. A series of dynamic triaxial tests, tests of voids in coarse aggregate, and X-ray CT scans were carried out on high plasticity soil-gravel mixtures of different gravel contents and gravel gradation types. The test results show that there is a critical gravel content, that is, 44.1%. When the gravel content is less than 44.1%, the mixture shows a dense suspended structure and its modulus increases slowly with increasing gravel content. When the gravel content is greater than 44.1%, the mixture exhibits a dense skeleton structure and the modulus increases rapidly as the gravel content rises. Moreover, as the gravel gradation tends to the lower type, coarse aggregates increase in quantity and contact each other to form a dense skeleton; thus, the modulus increases accordingly. As the gravel gradation approaches the upper type, coarse aggregates decrease in quantity and tend to suspend in the soil, so the modulus decreases. With the increase in contact number, the skeleton structure is continuously improved, and thus the modulus is enhanced progressively. The results indicate that the gravel mixing method with a gravel content of 40%–45% can effectively improve high plasticity soil and shows great environmental and economic benefits.