A new simple and effective measure for bag-of-word inter-document similarity measurement

Abstract

To measure the similarity of two documents in the bag-of-words (BoW) vector representation, different term weighting schemes are used to improve the performance of cosine similarity-the most widely used inter-document similarity measure in text mining. In this paper, we identify the shortcomings of the underlying assumptions of term weighting in the inter-document similarity measurement task; and provide a more fit-to-the-purpose alternative. Based on this new assumption, we introduce a new simple but effective similarity measure which does not require explicit term weighting. The proposed measure employs a more nuanced probabilistic approach than those used in term weighting to measure the similarity of two documents w.r.t each term occurring in the two documents. Our empirical comparison with the existing similarity measures using different term weighting schemes shows that the new measure produces (i) better results in the binary BoW representation; and (ii) competitive and more consistent results in the term-frequency-based BoW representation.

Dynamic Modeling of User Preferences for Stable Recommendations

“…For this task, we calculate similarities based on the Weighted Jaccard Index (WJI) [6], using the analogy with the task of finding weighted similarity between the text documents, as described in e.g. [4]. We encode recommendation lists as "bag-of-items" (BOI) vectors x = (𝑥 𝑖 ) 𝑁 𝑖=1 , where 𝑥 𝑖 denotes the rank (position in the list) of item 𝑖 and 𝑁 is the size of entire item catalog.…”