This paper presents a novel multi-scale template matching method that can be applied in unconstrained environments. The key component behind this is a general similarity measure is referred to as the diversity similarity measure against scaling, rotation, and illumination (DS-SRI). Specifically, DS-SRI exploits bidirectional diversity calculated from the nearest neighbour matches between two sets of points. Scaling and rotation changes are taken into consideration by introducing normalisation term on the scale change, and geometric consistency term with respect to the polar coordinate system. Moreover, in order to deal with the illumination change and further deformation, illumination-corrected local appearance and rank information are jointly exploited during the nearest neighbour search. All the features of DS-SRI are statistically assessed, and the extensive visual and quantitative results on both synthetic and real-world data show that DS-SRI can significantly outperform state-of-the-art methods.