The high influence of impact and vibration on the behavior of crushed stone and ballast materials has been known for a long time. The zones with unsupported sleepers, which are always present in transition zones, crossings, welds, etc., are typically characterized by impact interaction, ballast full unloading, and additional preloading. However, no studies on ballast layer settlements consider impact vibration loading. Moreover, the influence of the cyclic loading on the ballast settlement intensity is considered ambiguously, with both decelerating and accelerating trends. The comprehensive literature review presents the influence of factors on settlement intensity. The present study aims to estimate the long-term processes of sleeper settlement accumulation depending on the loading factors: impact, cyclic loading, and preloading. The typical for a void zone ballast loading pattern was determined for various void sizes and the position along the track by using a model of vehicle-track interaction that was validated by experimental measurements. The loading patterns were parametrized with four parameters: maxima of the cyclic loading, impact loading, sleeper acceleration, and minimal preloading. A specially prepared DEM simulation model was used to estimate the ballast settlement intensity after initial settlement stabilization for more than 100 loading patterns of the void zone cases. The settlement simulation results clearly show that even a low-impact loading pattern causes many times increased settlement intensity than ordinary cyclic loading. Moreover, the initial preloading in the neighbor-to-void zones can cause even a decrease in the settlement intensity compared to the full ordinary or partial unloading. A statistical analysis using a machine learning approach and an analytic one was used to create the model for the intensity prediction regarding the loading patterns. The analytic approach demonstrates somewhat lower prediction quality, but it allows to receive plausible and simple analytic equations of the settlement intensity. The results show that the maximal cyclic loading has a nonlinear influence on the settlement intensity that corresponds to the 3–4 power function, and the impact loading is expressed by the linear to parabolic function. The ballast’s minimal preloading contributes to the reduction of the settlement intensity, especially for high cyclic loadings that are typical for neighbor-to-void zones. The results of the present study could be used for the complementing of the present phenomenological equations with the new factors and further application in the algorithms of the settlements accumulation prediction.