The rainfall conditions cause seepage in the soil-rock mixture (SRM) filler subgrade, leading to the loss of fine particles and the change of soil structure, which eventually leads to large uneven deformation or instability of the subgrade. The particle loss test device was applied to conduct the seepage test of SRM filler, monitor the change process of permeable quality, fine particle loss and sedimentation of filler under different rainfall conditions, and analyze the evolution process of soil structure and sedimentation characteristics. The result shows that the rainfall intensity affects the permeability of the filler, and then accelerates the loss of fine particles under water migration. With the condition of the same rainfall duration, the hourly water permeability increased firstly and then gradually stabilized with the rainfall duration. The total water permeability mass, fine particle loss, and real-time sedimentation increased with the rainfall intensity. With the same rainfall condition, the maximum hourly water permeability under short-term heavy rainfall condition is about three times that under heavy rainfall condition, which is more serious for the internal erosion of SRM filler. The total water permeability mass, fine particle loss, and real-time sedimentation increased with the rainfall intensity. The stages of skeleton remodeling and relative stability are the most serious stages of filler skeleton structure damage. The sedimentation deformation of filler has hysteresis, resulting in the occurrence of sedimentation much later than the loss of fine particles and skeleton deformation. After the rainfall stops, with the loss of fine particles and water dissipation, the filler will occur secondary sedimentation, resulting in an increase in the final sedimentation as the rainfall intensity increases, making it possible for subsidence damage of the roadbed to occur both during and after the rainfall. Extreme rainfall conditions (short-term heavy rainfall) have the most obvious effect on the structural damage of the subgrade.