A Close-to-linear Topic Detection Algorithm using Relative Entropy based Relevance Model and Inverted Indices Retrieval

Abstract

Although timely access to information is becoming increasingly important and gaining such access is no longer a problem, the capacity for humans to assimilate such huge amounts of information is limited. Topic Detection(TD) is then a promising research area that addresses speedy access of desired information. However, ironically, the time complexity of existing TD algorithms themselves is usually O(n 3 ) or up to the x-th power of e. Linear performance requirement of real world topic detection has not been significantly addressed. This paper reveals a new patented topic detection algorithm called RMIR that combines relevance model with information retrieval technique to improve on time efficiency. Relevance Model(RM) is a theoretical extension of statistical language modeling that was developed for the task of document retrieval. To reduce the costs of fetching RM, we reduce the number of comparisons for stories by a query-based approach that makes similar stories exist in the top-k query results. We also build our query based on inverted indices, which have the complexity close to linear. The time cost of rest of operations in the RMIR topic detection process is a constant. Hence, the total complexity of RMIR topic detection algorithm should be close to linear as shown in experimental results. In addition, RMIR also gains better detection rates and robustness by relative entropy based topic model design.

Quality and Safety News Topic Tracking Algorithm Based on Improved K-Nearest Neighbor

“…Xu (2014) gave three topic tracking models: a static topic model BSTM and two dynamic topic models BDTM-I, and BDTM-II by using Bayesian belief network [5]. The hidden topic detection algorithm based on related model retrieval technology was proposed by Shi (2012) [6]. Kumaran (2004), Nallapati (2004) introduced the natural language processing technology into the topic detection, and it is verified that nature language technology can improve topic detection quality effectively [7][8].…”

Personalized Subject Learning Based on Topic Detection and Canonical Correlation Analysis

“…Such a TD model design facilitates code design that then achieves linear performance with the combination of full text retrieval and new algorithm as shown in [16]. Other TD algorithms reported in literature have non-linear performance.…”

“…The experiments tested the viability of our work, in the context of real time fresh online and offline contents of NIST. Detection rate is justified by means of link detection task (LDT) as stated in [16].…”

“…Based on previous research [12,13,14,15,16,17,18], a novel personalized subject learning (PSL) system is created based on the above ideas. PSL system is a computer aided education system using TD technologies and CCA methodology.…”