The backfill area of tunnel projects may deform or collapse due to the cyclic disturbance of groundwater and train loads. Hence, the anti-deformation and crack resistance performance of backfill materials under cyclic disturbance is critical to engineering safety. In this paper, concrete was produced by mixing 0.85 mm, 1–3 mm and 3–6 mm rubber particles instead of 10% sand, and tested to discuss the effect of rubber particle size on the deterioration of concrete material properties (compressive characteristics and energy dissipation) after bearing cyclic loading. The stress–strain curve and various parameters obtained through the uniaxial compression test and cyclic load test were used to explore the optimal grain size that can be applied to the tunnel engineering backfill area, and numerical simulation was adopted to calculate the deformation of the surrounding rock and the structural stress of different backfill materials. Research shows that the increase in particle size lessens the compressive strength, deformation resistance and cracking resistance of specimens, but after the cyclic loading test, the concrete material deterioration analysis indicates that rubber concrete has lesser and more stable losses compared to ordinary concrete, so the optimum rubber particle size is 0.85 mm. Numerical calculations show that RC-1 reduces the arch top displacement by 0.4 mm, increases the arch bottom displacement by 0.6 mm and increases the maximum principal stress by 11.5% compared to OC. Therefore, rubber concrete can ensure the strength and stability requirements of tunnel structures, which can provide a reference for similar projects.