Vibrating machinery with an elliptical trajectory widely exists. The bulk particles show a regular sliding motion and throwing motion under the periodic vibration of the frame. It is essential for the theoretical design and engineering application of such machinery to study the particle's motion theory based on considering the interactions between particles and frame in detail. In the first, a time-varying piecewise nonlinear dynamic model of the system is established based on the force analysis of the particles in the state of sliding and throwing motion. The dimensionless equations of systems with nonlinear terms are deduced. After that, the feasibility of using a single particle to study the particle flow's movement is verified by the discrete element method. Then, the system's approximate steady-state analytical solution is obtained by the incremental harmonic balance method. Its numerical solutions are solved by the Newmark-β method, and the solutions obtained by the three methods are verified mutually. In addition, the Sommerfeld effect in vibration system and the system's related parameters are analyzed. Finally, an experimental platform of particles' movement is built. The feasibility of the theoretical analysis in this article was verified by particle's movement experiments.