Efficient jaccard-based diversity analysis of large document collections

Abstract

We propose two efficient algorithms for exploring topic diversity in large document corpora such as user generated content on the social web, bibliographic data, or other web repositories. Analyzing diversity is useful for obtaining insights into knowledge evolution, trends, periodicities, and topic heterogeneity of such collections. Calculating diversity statistics requires averaging over the similarity of all object pairs, which, for large corpora, is prohibitive from a computational point of view. Our proposed algorithms overcome the quadratic complexity of the average pair-wise similarity computation, and allow for constant time (depending on dataset properties) or linear time approximation with probabilistic guarantees. Our theoretical findings are verified on synthetic and real-world data sets. As applications, we show examples of diversity-based studies on large samples from corpora such as the social photo sharing site Flickr, the DBLP bibliography, and US Census data.

Classification of Alzheimer’s Disease and Mild Cognitive Impairment Based on Cortical and Subcortical Features from MRI T1 Brain Images Utilizing Four Different Types of Datasets

“…The kappa can range between −1 and + 1, and the kappa result is explained as follows: if the values are ≤0, there is no agreement; between 0.01 and 0.20, there is minor arrangement; between 0.21 and 0.40, there is known fair agreement; between 0.41 and 0.60, there is moderate agreement, between 0.61 and 0.80, there is substantial agreement; and from 0.81 to 1.00, there is nearly perfect agreement [ 29 ]. The Jaccard index determines how close the commonality of the two datasets can be a measured [ 30 ]. The Jaccard coefficient is given in the following equation: …”

Building relatedness explanations from knowledge graphs

“…The above definition is used to compute the pairwise diversity of a set of paths P. This in general requires O(|P| 2 ) computations. To avoid pairwise computations, one can use min-wise hashing [32].…”

Continuous Similarity Computation over Streaming Graphs

“…To express the similarity between graph nodes, a meaningful similarity measure is required. One such measure is the Jaccard similarity, which has been applied successfully in areas such as duplicate detection [6,19], link prediction [15], similarity evaluation in wikipedia [4], triangle counting in massive graphs [5] and diversity analysis in documents [9].…”

“…In general, similarity is expressed by a function V ×V → [0,1], where a value close to 0 means low similarity and a value close to 1 denotes a high similarity between a node pair. In this work, we express similarity by means of the Jaccard similarity coefficient, which enjoys a widespread use in diverse areas such as link prediction and recommendation [15], data cleaning [3], near duplicate detection [19], diversity analysis [9], whereas it is one of the most important measures for set similarity. We associate with each node u the set of its immediate neighbors N (u) (u inclusive).…”