Impact loads generated from the dynamic effect of passing trains can exacerbate the degradation level of ballast aggregate of railway track. To diminish the induced impact loads, the use of crumb rubber (CR) in the ballast course is characterized as a well-established procedure related to the modification of utilized material. Nonetheless, more in-depth assessments of size and percentage of CR particles combined with ballast aggregate are still required. The present study evaluates the influence of size and content of CR particles used for degradation reduction of ballast aggregate subjected to impact loading. For this purpose, a large-scale impact loading test is carried out on prepared specimens of aggregate by considering the initial gradation, subgrade condition, as well as the size and content of CR particles. The results indicate less ballast degradation for a higher percentage of CR particles. Meanwhile, the enhancement of rubber modified ballast against deterioration is further highlighted in the case of rigid subgrade. In addition, incorporation of larger-sized CR particles (12.5–25 mm) in a ballast specimen comprising more uniform gradation of aggregate can more effectively diminish the amount of degradation. Nevertheless, using smaller-sized CR particles (4.75–9.5 mm) for a ballast sample consisting of a broader range of sizes can better improve resistance against degradation.