Ballast is coarse aggregate with particle size normally ranging from 10 mm to 65 mm. Upon repeated train loading, ballast deteriorates in the form of either continuous abrasion of sharp corners or size degradation, which have been reported as the fundamental cause for the instability of railway tracks. In this study, the splitting behavior of ballast grain with varying particle sizes under diametrical compression was examined to investigate the size effect and the Weibull characteristics of ballast tensile strength; a Weibull modulus of 2.35 was measured for the tested granite ballast. A series of large-scale monotonic triaxial tests on ballast aggregates having various size gradings was performed to study the effect of particle gradation on the mechanical behavior of ballast. The results show that compared to mono-sized uniformly distributed aggregates, non-uniformly distributed aggregates generally have greater shear strength, larger peak friction angle, 50% strength modulus, and greater volumetric dilation. The ballast aggregate conforming to the recommended PSD as per current standards exhibited the most superior mechanical performance, possessing the greatest shear strength, peak friction angle, and 50% strength modulus. Micromechanical analysis showed that aggregates with larger d50 values have higher coordination numbers, inter-particle contact forces, and higher anisotropy level of contact normals, thus causing a greater possibility of particle breakage during shearing.