A soil–rock mixture (SRM) is a type of heterogeneous geomaterial, and the particle distribution of SRM can be described by fractal theory. At present, it is difficult to quantify the fractal dimension of a particle size distribution and understand the scale effect in SRMs. In this study, the fractal theory and discrete element method (DEM) were introduced to solve this problem. First, the particle gradation of SRM was dealt with by using fractal theory. The fractal structure of particle distribution was studied, and a method of calculation of the fractal dimension is presented in this paper. Second, based on the fractal dimension and relative threshold, the particle gradations of SRMs at different scales were predicted. Third, numerical direct shear tests of SRM at different scales were simulated by using the DEM. The scale effects of shear displacement, shear zone, and shear strength parameters were revealed. Last, taking the maximum particle size of 60 mm as the standard value, the piece-wise functional relationship between shear strength parameters and particle size was established. The results are as follows: for SRM in a representative engineering area, by plotting the relationship between particle cumulative mass percentage and particle size, we can judge whether the SRM has a fractal structure; in Southwest China, the frequency of the fractal dimension of the SRM is in the normal distribution, and the median fractal dimension is 2.62; the particle gradations of SRMs at different scales calculated by fractal dimension and relative threshold can expand the study scope of particle size analysis; when the particle size is less than 70 mm, the strength parameters show a parabolic trend with the particle size increases, and if not, a nearly linear trend is found. The proposed method can describe the fractal characteristics of SRM in a representative engineering area and provides a quantitative estimation of shear strength parameters of SRM at different scales.