Completely weathered phyllite (CWP) soil is a kind of special soil with high swell potential, while red clay is a special soil with high shrinkage. This means that these two kinds of special soils are usually not suitable for direct use as subgrade fill. To reduce the swell index of the CWP soil and the shrinkage of red clay at the same time, it was proposed to blend the CWP soil with red clay to improve their basic characteristics. A series of swell index tests and dry-wet cycle tests of the blended soils have been carried out at varying blending ratios, compaction coefficients and moisture contents. The test results show that the free swell index of the blended soil decreases with the increase of red clay, moisture content and compaction coefficient, respectively. The fissure density of the blended soil first decreases and then increases with the blending ratio, with the lowest being zero when the blending ratio is ranging from 20% to 40%. Through particle microscopic analysis and elemental composition analysis, it is found that the neutralization effect, the dilution effect of swell minerals, and the partition effect of coarse particles play an important role in restraining expansion and shrinkage deformation of the blended soil. Based on the liquid limit requirement of Chinese Railway Design Code (TB 10001-2016), the optimal blending ratio of red clay has been proposed to be 50%. Compared with the CWP soil, the free load swell index of the blended soil is reduced by 45.0% and the fissure density is reduced by 99.3% compared with that of red clay. Therefore, it is feasible to improve the CWP soil by blending it with red clay at an optimal ratio of 50% by using the neutralization effect of the expansion of CWP and shrinkage of red clay.