To determine the energy attenuation and energy reflection coefficients in layered foundation is the key factor to reveal the dynamic response characteristics of the high-speed railway foundation. Based on the foundation test model under the dynamic loading of the high-speed railway, the energy attenuation and energy reflection coefficients were introduced and the attenuation formulas of the vibration acceleration in the layered foundation were deduced. Five scale models of 1:1, 1:2, 1:5, 1:10 and 1:20 are established respectively by using Abaqus technique. The energy attenuation mechanism and interfacial energy reflection characteristics in the layered foundation were analyzed. Results show that it is appropriate to use vibration acceleration to characterize the propagation rule of the energy attenuation in the layered foundation. The size effect equations of the energy attenuation and energy reflection coefficients in the layered foundation are deduced and the size effect of the energy attenuation is revealed. Based on the nonlinear relationships among the energy attenuation coefficient, the model scale, the loading amplitude and the vibration frequency, the energy attenuation equation of the scale test model is constructed. The reliability of the theoretical and simulation results is verified by the scale test model. The conclusions obtained in this study can provide a reference for a similar engineering practice.