Are Topically Diverse Documents Also Interesting?

Abstract

Text interestingness is a measure of assessing the quality of documents from users' perspective which shows their willingness to read a document. Different approaches are proposed for measuring the interestingness of texts. Most of these approaches suppose that interesting texts are also topically diverse and estimate interestingness using topical diversity. In this paper, we investigate the relation between interestingness and topical diversity. We do this on the Dutch and Canadian parliamentary proceedings. We apply an existing measure of interestingness, which is based on structural properties of the proceedings (eg, how much interaction there is between speakers in a debate). We then compute the correlation between this measure of interestingness and topical diversity.Our main findings are that in general there is a relatively low correlation between interestingness and topical diversity; that there are two extreme categories of documents: highly interesting, but hardly diverse (focused interesting documents) and highly diverse but not interesting documents. When we remove these two extreme types of documents there is a positive correlation between interestingness and diversity.

HiTR: Hierarchical Topic Model Re-Estimation for Measuring Topical Diversity of Documents

“…Each entry of this vector corresponds to a document d y and its value is assigned as: V x [y] = p y x . We use the normalized angular distance for measuring the distance between topics, since it is a proper distance function [2]:…”

“…Q Uantitative notions of measuring topical diversity of text documents are useful in a number of applications, such as assessing the interdisciplinariness of a research proposal [1] and helping to determine the interestingness of a document [2][3][4].…”

“…Our main contributions are: (1) We propose a hierarchical re-estimation process for topic models to address the two main problems in estimating the topical diversity of text documents, using a biologically inspired definition of diversity. (2) We study each level of re-estimation individually in terms of efficacy in solving the general topics problem, the impure topics problem, and improving the accuracy of estimating the topical diversity of documents. (3) We study the impact of re-estimation parameters on the statistics of documents and its relation to the quality of trained topic models and recommend effective settings of these parameters.…”

Measuring Interestingness of Political Documents

“…For all other uses, contact the owner/author(s). debates [1] or the structural information of debates [6]. Both of these approaches do not use the main content of the debates for assessing their interestingness.…”

Hierarchical Re-estimation of Topic Models for Measuring Topical Diversity

“…Quantitative notions of topical diversity in text documents are useful in several contexts, e.g., to assess the interdisciplinarity of a research proposal [3] or to determine the interestingness of a document [2]. An influential formalization of diversity has been introduced in biology [17].…”