Robust methods based on nonlinear influence functions are often used to remove outliers from data. The article describes the design of an algorithm for tracking a moving object in a thermal image using a SURF descriptor and robust Kalman filter. However, given that the main shortfall of robust methods is reduced efficiency, tunable parameters of the robust influence function are used to achieve a balance between robustness and efficiency. The parameters adapt to the observed scene's current conditions, while their estimation is based on the calculated data outlier contamination factor. The outliers that contaminate the data are generally a result of various types of occlusions. The results show that the proposed solution achieves better performance than the standard Kalman filter or fixed-parameter robust Kalman filter.