The presence of unknown heavy-tailed noise can lead to inaccuracies in measurements and processes, resulting in instability in nonlinear systems. Various estimation methods for heavy-tailed noise exist. However, these methods often trade estimation accuracy for algorithm complexity and parameter sensitivity. To tackle this challenge, we introduced an improved variational Bayesian (VB)-based adaptive iterative extended Kalman filter. In this VB framework, the inverse Wishart distributionis used as the prior for the state prediction covariance matrix. The system state and noise parameter posterior distributions are then iteratively updated for adaptive estimation. Furthermore, we make adaptive adjustments to the IEKF filter parameters to enhance sensitivity and filtering accuracy, thus ensuring robust prediction estimation. A two-dimensional target tracking and nonlinear numerical UNGM simulation validated our algorithm. Compared to existing algorithms RKF-ML and GA-VB, our method showed significant improvements in RMSEpos and RMSEvel, with increases of 21.81% and 22.11% respectively, and a 49.04% faster convergence speed. These results highlight the method’s reliability and adaptability.