A well-known problem of visual object tracking is the difficulty of accurately estimating the object trajectory under conditions of environmental disturbances in the bounding box (BB) of a video camera. In this paper, we consider BB variations as Gaussian-Markov colored measurement noise (CMN). In order to perform accurate tracking in the presence of CMN, we use measurement differencing and develop a robust general unbiased finite impulse response (GUFIR) filter and use the general Kalman filter (GKF) as a benchmark. The GUFIR and GKF algorithms are tested by the ''Car4'' benchmark. It is shown that, in terms of the tracking precision and under the heavy disturbance with the 0.65 ⩽ ⩽ 0.95 coloredness factor, the best tracking performance is achieved using the robust GUFIR filter. When < 0.6, the GUFIR and GKF algorithms perform near equally. In the extreme point of = 1.0, where the Gauss-Markov CMN loses the stationarity, both algorithms provide zero precision and become inefficient. In general, it is concluded that the GUFIR filter, which ignores any zero mean disturbance and initial values, is much more suitable for applications in visual object tracking than Kalman-like algorithms relying on complete object information.INDEX TERMS Visual object tracking, bounding box, environmental disturbance, colored measurement noise, general UFIR filter, general Kalman filter, precision.