Image registration is an important research topic in the field of computer vision. Traditional non-rigid image registration methods are based on the correctly matched corresponding landmarks, which usually needs artificial markers. It is a rather challenging and demanding task to locate the accurate position of the points and get the correspondance. In order to get the most correctly matched point set automatically, a new point matching method based on deformation invariant feature and local affine-invariant geometric constraint is proposed in this paper. Particularly mention should be the geodesic-intensity histogram (GIH), an interesting deformation invariant descriptor, which is introduced to describe the local feature of a point. In addition, the local affine invariant structure is employed as a geometric constraint. Therefore, an objective function that combines both local features and geometric constraint is formulated and computed by linear programming efficiently. Then, the correspondence is obtained and thin-plate spline (TPS) is employed for non-rigid registration. Our method is demonstrated with deliberately designed synthetic data and real data and the proposed method can better improve the accuracy as compared to the traditional registration techniques.