The physical parameters (stiffness, damping) of time-varying (TV) systems under random excitation provide valuable information for their working condition but they are often overwhelmed by noise interference. To overcome this problem, this paper presents a novel multi-level kernel regression residual decomposition method, which can not only effectively separate each modal component from the raw vibration acceleration signal, but also eliminate noise interference. Additionally, the multiple degree-of-freedom (DOF) parameter identification problem is transformed into a single DOF parameter identification problem. Combined with the derived polynomial frequency modulation integral algorithm and the cross-correlation theory based on the fractional Fourier ambiguity function, a physical parameter identification method is proposed. The method provides a new idea in modeling TV systems and identifying physical parameters under random excitation. To demonstrate the effectiveness of the proposed method, numerical simulations are conducted with three different cases of variation (variation, quadratic variation, and periodic variation) in time. Moreover, its robustness is evaluated by adding different signal-to-noise ratio levels of noise (20 dB, 50 dB, 100 dB) to the input vibration acceleration signal. The analysis results confirm the performance of the proposed method for the parameter identification of TV systems under random excitation.