To measure the similarity between two images, the local information around all the points were usually used for traditional image retrieval methods based on interest points. The accuracy would be influenced by the dissimilar interest points in the background region or in the uninterested regions. A new image retrieval method based on interest points was proposed in this paper. The most similar interest points were chosen firstly based on feature matching techniques, then the similarity was measured based on the local gray information around the preserved similar interest points. Experiments show that the proposed method is more accurate than traditional image retrieval method based on interest points.