Re-evaluating Word Mover's Distance

Abstract

The word mover's distance (WMD) is a fundamental technique for measuring the similarity of two documents. As the crux of WMD, it can take advantage of the underlying geometry of the word space by employing an optimal transport formulation. The original study on WMD reported that WMD outperforms classical baselines such as bag-of-words (BOW) and TF-IDF by significant margins in various datasets. In this paper, we point out that the evaluation in the original study could be misleading. We re-evaluate the performances of WMD and the classical baselines and find that the classical baselines are competitive with WMD if we employ an appropriate preprocessing, i.e., L1 normalization. However, this result is not intuitive. WMD should be superior to BOW because WMD can take the underlying geometry into account, whereas BOW cannot. Our analysis shows that this is due to the high-dimensional nature of the underlying metric. We find that WMD in high-dimensional spaces behaves more similarly to BOW than in low-dimensional spaces due to the curse of dimensionality.Preprint. Under review.

Moving Other Way: Exploring Word Mover Distance Extensions

“…(Wang et al, 2020) replace assumption that documents' BOWs have the same measure to solve Kantorovich problem of optimal transport with the usage of Wasserstein-Fisher-Rao distance between documents based on unbalanced optimal transport principles. The work of (Sato et al, 2021) is especially significant for further discussion. The authors re-evaluate the performances of WMD and the classical baselines and find that once the data gets L1 or L2 normalization, the performance of other classical semantic similarity measures becomes comparable with WMD.…”

“…To assure better reproducibility, we work with the datasets presented in (Kusner et al, 2015) and (Sato et al, 2021) 1 . For the evaluation, we use six datasets that we believe to be diverse and illustrative enough for the aims of this discussion.…”

“…Similar to (Sato et al, 2021), we split the training set into an 80/20 train/validation set and select the neighborhood size from [1, 20] using the validation dataset.…”

“…• vanilla WMD • WMD with TF-IDF regularization applied to bags of words for both documents (Sato et al, 2021) • WRD -Word Rotator's Distance (Yokoi et al, 2020) to compute it authors use 1 − cos(w i , w j ) as a distance or transportation cost between words w i and w j . They multiply the document's BOW by words norm.…”

A matching method of oral text to instruction based on word vector

“…In 2021, researchers such as Ryoma Sato conducted a more in-depth study of WMD and found that the evaluation method in the original study of WMD has certain errors. The similarity calculation can be made more accurate if appropriate preprocessing, namely L1 normalization, is used [7] .…”