This article uses a combined pendulum mode to study the segregation behavior of spherical particles of pyrite and brass ore with different diameters using the discrete element method. The numerical simulation program includes extensive oscillation tests on binary mixed particles of pyrite and brass ore spherical particles in different steel U-shaped oscillation grooves, different oscillation frequencies, oscillation angles, and particle sizes. The dispersion coefficient γ calculated from the Lacey mixing index M is used to characterize the segregation behavior of binary mixed particles and track the evolution of segregation. Numerical simulation shows that the oscillation frequency is 8 Hz, the oscillation amplitude is 14°, and the dispersion coefficient γ of the first group with protrusions reaches 0.8, which is about 19.4% higher than the dispersion coefficient 0.67 of the first group with light grooves. The dispersion coefficient γ of the second group with added protrusions reached 0.69, which is about 16.95% higher than the dispersion coefficient 0.59 of the second group when using a light groove. The addition of protrusions improved the particle layering effect. The oscillation amplitude is taken as 14°, and the oscillation frequencies are taken as 6Hz, 8Hz, 10Hz, and 12Hz, respectively. As the oscillation frequency increases, collisions between particles become more intense, and the dispersion coefficient γ between particles increases to a certain extent when it changes from 6Hz to 8Hz. Then, as the oscillation frequency increases, the dispersion coefficient γ decreases in severe fluctuations, and the particle stratification effect actually deteriorates. The oscillation frequency is taken as 8Hz, and the oscillation amplitude is taken as 10°, 14°, and 18°, respectively. After the oscillation amplitude increases to 14°, the dispersion coefficient γ of the particles remains basically stable, and changing the oscillation amplitude has almost no effect on the particle stratification effect. The oscillation frequency is 8Hz, and the oscillation amplitude is 14°. When adding protrusions to the steel U-shaped oscillation groove, the average relative errors of both experimental schemes are less than 1%, and the correlation coefficients are close to 1. The data points are very close to the 45° line. It indicates that the experimental data and simulation values have high consistency, and the discrete element method is feasible and reliable as a method for studying mineral particulate matter. The in-depth study of the separation and stratification behavior of ore particles under combined oscillation is of great scientific significance for the development of new combined vibration equipment.