The angle of repose (AOR) is one of the key parameters used to comprehensively characterize the basic mechanical properties and stacking properties of granular materials. In this research, the influence of temperature and moisture on the stacking performance of railway ballast was studied through tests of the AOR. Additionally, the parameters of the discrete element method (DEM) for ballast beds with various temperatures and moisture levels were calibrated and combined with the response surface method. Then, the AORs of the ballast in different environments were simulated with the DEM, and the micromechanical properties of the ballast in dry, wet, low-temperature, and frozen environments were compared. The results showed that the AOR of the ballast in a frozen environment was smaller than that in a dry environment at the same low temperature (the difference could reach 8.48° at −30°C). The AOR of the ballast increased with the decrease of the temperature in a dry environment (18.44% increase between 20°C and −30°C). The AOR of the ballast decreased with the decrease of the temperature in a wet (frozen) environment (22.86% decrease between 20°C and −30°C). In addition, the validity tests of the AOR simulations of ballast proved that the obtained parameters could be used in the DEM models of ballast for the corresponding temperature and moisture. At the same time, it was found that with the change of temperature and moisture, the force chain network of the ballast would also change.