Conventional approaches to object tracking use area correlation, but they are difficult to solve the problem of deformation of object region during tracking. A novel target tracking method based on Lie algebra is presented. We use Gabor feature as target token, model deformation using affine Lie group, and optimize parameters directly on manifold, which can be solved by exponential mapping between Lie Group and its Lie algebra. We analyze the essence of our method and test the algorithm using real image sequences. The experimental results demonstrate that Lie algebra method outperforms other traditional algorithms in efficiency, stabilization and accuracy.