Video stabilization is a technology to remove the dithering between frames in a video sequence by motion estimation, motion filter and motion compensation. In this paper, we focus on the motion estimation (ME) step which is known as the most important part for a successful video stabilization system because the latter processing heavily relies on it. However, traditional ME methods mainly hypothesize that the movement within frames only consists of displacement. So their methods will lose their validity on the occasion when rotation is also presented. To solve this problem, interest points matching (IPM) was often suggested as suitable solution. But to determine the parameter of interest point and to get a reliable match result involve a lot of computation load. Hence, we proposed a novel motion estimation scheme which uses edge points as interest points and then follows a coarse-to-fine matching method to obtain the motion parameter. Experiment shows our method can achieve a robust performance under various environments and the processing time is less than traditional corners based IPM methods.