Video synthetic aperture radar (ViSAR) has been found very useful for the surveillance of ground moving targets. The target energy can be utilized for ground moving target tracking, while the dynamic shadows of moving targets enable an alternative tracking approach. However, neither of these two approaches can stand alone to provide reliable target tracking. The smeared shadow and energy both degrade the tracking performance when the target is maneuvering.A moving target tracking framework based on the joint kernelized correlation filter (JKCF) has been developed. Based on the feature training of JKCF, the target is tracked by combining its shadow in the sequential SAR imagery and the corresponding energy in the range-Doppler (RD) spectra. Aiming at the problems of tracking drift and collapse, interactive processing is adopted to enhance the target positioning and feature update based on the confidence assessment. By cooperating with the initialization and feature update strategy, the tracking success rate and precision can be improved significantly.