In general, the extended Kalman filter (EKF) has a wide range of applications, aiming to minimize symmetric loss function (mean square error) and improve the accuracy and efficiency of state estimation. As the nonlinear model complexity increases, rounding errors gradually amplify, leading to performance degradation. After multiple iterations, divergence may occur. The traditional extended Kalman filter cannot accurately estimate the nonlinear model, and these errors still have an impact on the accuracy. To improve the filtering performance of the extended Kalman filter (EKF), this paper proposes a new extended Kalman filter (REKF) method that utilizes the statistical properties of the rounding error to enhance the estimation accuracy. After establishing the state model and measurement model, the residual term is used to replace the higher-order term in the Taylor expansion, and the least squares method is applied to identify the residual term step by step. Then, the iterative process of updating the extended Kalman filter is carried out. Within the Kalman filter framework, a higher-order rounding error-based extended Kalman filter (REKF) is designed for the joint estimation of rounding error and random variables, and the solution method for the rounding error is considered for the multilevel approximation of the original function. Through numerical simulations on a general nonlinear model, the higher-order rounding error-based extended Kalman filter (REKF) achieves better estimation results than the extended Kalman filter (EKF) and improves the filtering accuracy by utilizing the higher-order rounding error information, which also proves the effectiveness of the proposed method.