In this paper, we propose a robust model for tracking in video sequences with non-static backgrounds. The object boundaries are tracked on each frame of the sequence by minimizing an energy functional that combines region, boundary and shape information. The region information is formulated by minimizing the symmetric KullbackLeibler (KL) distance between the local and global statistics of the objects versus the background. The boundary information is formulated using a color and texture edge map of the video frames. The shape information is calculated adaptively to the dynamic of the moving objects and permits tracking that is robust to background distractions and occlusions. Minimization of the energy functional is implemented using the level set method. We show the effectiveness of the approach for object tracking in color, infrared (IR), and fused colorinfrared sequences.