Railway ballasted track stiffness is an important indicator to identify supporting condition that ensures the facility is well designed and functioned. Although many studies have been performed on track stiffness based on experimental tests and finite element methods, the factors influencing the track stiffness have not been completely confirmed yet, especially the influences from ballast and subgrade layers at a mesoscopic level. To address this research gap, a coupled the discrete element method (DEM) and the finite difference method (FDM) model is utilised to study the factors influencing on the track stiffness from the particle level. Factors (related to ballast layer properties) are bulk density, thickness and stiffness, and other factor (related to subgrade properties) is elastic modulus. Additionally, the relationship between the track stiffness and the mechanical behaviour of ballast is analysed. This study quantified the influences of track components on the track stiffness and accordingly proposed how to improve it from the ballast and subgrade layers at the mesoscopic level, which can provide the guidance for railway ballasted track design and maintenance.