The Anqing group clay gravel layer is a special geological body composed of gravel and clay. In excavation projects, involving this soil, such a gravel layer, is prone to slope collapse and instability under the influence of rainfall. To clearly understand the failure mechanism and influencing factors of clay gravel slopes, an indoor artificial rainfall erosion model testing was carried out to analyse the effect of various slope ratios, gravel contents, and rainfall intensities. The slope erosion damage form, runoff rate, infiltration rate, scoured material, and slope stability of the clay gravel slope were studied. The test results show that sloping surfaces of the gentle slope were mainly damaged by erosion, and the degree of damage gradually increased from the top to the bottom of the sloping surface; however, the stability of the surface was good. In the case of the sloping surface layer of the steep slope, large-scale landslides occurred, and the stability of the surface was poor. When the gravel content was small, the surface failure was manifested as a gully failure. When the gravel content was large, it was manifested as a “layer-by-layer sliding” failure. The degree of influence of different conditions on the stable runoff rate was as follows: rainfall intensity>slope ratio>gravel content. The degree of influence of the parameters on the stable infiltration rate was as follows: slope ratio>rainfall intensity>gravel content. On gentle slopes, the total mass of the scoured material was inversely proportional to the gravel content and directly proportional to the rainfall intensity; on a steep slope, the total mass of the scoured material increased with an increase in the rainfall intensity and gravel content. Moreover, the slope ratio was the key influencing factor to decide whether there was gravel in the scoured material.