Best-Buddies Similarity—Robust Template Matching Using Mutual Nearest Neighbors

Abstract

Abstract-We propose a novel method for template matching in unconstrained environments. Its essence is the Best-Buddies Similarity (BBS), a useful, robust, and parameter-free similarity measure between two sets of points. BBS is based on counting the number of Best-Buddies Pairs (BBPs)-pairs of points in source and target sets, where each point is the nearest neighbor of the other. BBS has several key features that make it robust against complex geometric deformations and high levels of outliers, such as those arising from background clutter and occlusions. We study these properties, provide a statistical analysis that justifies them, and demonstrate the consistent success of BBS on a challenging real-world dataset while using different types of features.

Explaining away results in accurate and tolerant template matching

“…Another approach is to define alternative metrics for comparing the template with the image that allow for mis-alignment between the pixels in the template and the corresponding pixels in the image, rather than rigidly comparing pixels at corresponding locations in the template and the image. Typically, these metrics are based on measuring the distance between points in the template and the best matching points in the image (Dekel et al, 2015;Oron et al, 2018;Talmi et al, 2017). For example, the Best-Buddies Similarity (BBS) metric (Dekel et al, 2015;Oron et al, 2018) is computed by counting the proportion of sub-regions in the template and the image patch that are "best-buddies".…”

“…This dataset uses images that are 20 frames apart. A similar dataset with 25, 50, or 100 frames between pairs of images was used to test the BBS algorithm in (Oron et al, 2018). However, this alternative dataset has not been made publically available.…”

“…The overall accuracy of each method was summarised using the area under the success curve (AUC) for all 1000 template matches performed (25 templates matched to 5 images in each of 8 sequences). As the same Algorithm AUC patch size (pixels): 17-by-17 33-by-33 49-by-49 ZNCC 0.4996 0.5937 0.6314 BBS (Dekel et al, 2015;Oron et al, 2018) 0.1782 0.3834 0.4747 DDIS (Talmi et al, 2017) 0.3952 0.4905 0.5334 DIM (25 templates) 0.5591 0.6308 0.6569 Table 2: Quantitative comparison of results for different algorithms applied to the task of finding corresponding locations across images in the Oxford VGG affine covariant features dataset (at half size). Results are given in terms of the area under the success curve (AUC) for all 1000 template-image comparisons across all eight sequences in the dataset (25 templates per image pair).…”

“…Algorithm f-score patch size (pixels): 17-by-17 33-by-33 49-by-49 ZNCC 0.2842 0.5508 0.5493 BBS (Dekel et al, 2015;Oron et al, 2018) 0.0822 0.3704 0.4579 DDIS (Talmi et al, 2017) 0.5198 0.5355 0.4959 DIM (70 templates) 0.6542 0.7230 0.7513 magnitude such that true and false matches can not be reliably distinguished. The performance of DDIS was similar for all template sizes, but was only superior to that of ZNCC at the smallest size.…”

“…Using the Fourier method of calculating the cross-correlations and convolutions, the complexity of DIM would be approximately O(2(M + 2m)(N + 2n)log(M N )cti). Cross-correlation and con-Algorithm Execution Time (s) ZNCC 15 (0.14) 33 (0.03) 59 (0.02) BBS (Dekel et al, 2015;Oron et al, 2018Oron et al, ) 2193 724 (0.72) 1512 (0.62) DDIS (Talmi et al, 2017) 4802 ( Table 5: Comparison of the execution times of different algorithms. The first column of times show the total time taken when the algorithms where applied to the task of finding corresponding locations in 105 pairs of colour video frames (i.e., to obtain the results shown in fig.…”