Computing paper similarity based on latent dirichlet allocation

Evaluación de distintas técnicas de representación de texto y medidas de distancia de texto usando KNN para clasificación de documentos

“…The topic modelling technique known as LDA is used in [9,7] and considers two main concepts: 1) a single document can have several latent topics and 2) each topic can be drawn as a probability distribution of words (documents are represented as vectors of topics instead of bags of words). A supervised variant of LDA (sLDA) is proposed in [10], which incorporates a response variable (or class) when calculating the model of topics.…”

“…Regarding text distance metrics the literature is very extensive, being the Cosine similarity one of the most used techniques, like in [17,6]. Authors in [7] combine LDA representation with the Cosine function to calculate similarity among publications using the text of the title, abstract and author names. Other metrics used for both classification and grouping of text documents include simpler vector functions (Manhattan and Euclidean), set-based metrics (Jaccard) and entropy measure in probability distributions (KLD) [17,6,3].…”

“…Among the most common metrics used for words vector comparison are the Cosine and Jaccard similarities [6,3]. However, there are some approaches based on probabilistic concepts that have also been used for text documents comparison, such as KLD [7,3]. All these methods are explained in the below sections.…”

“…LDA is a probabilistic model used to classify documents in topics considering two aspects: 1) the same document can have several latent topics and 2) each topic can be represented by a distribution of words [7]. In this case, a documents will be assigned to probabilities of latent topics where, at the same time, each topic is a probability distribution of words.…”

“…Additionally, topic modelling has arisen as a promising alternative to the existing methods, by achieving good results in documents classification using probability distributions similarity, dimensional reduction and incorporating topics semantic [7,8,9]. Latent Dirichlet Allocation (LDA) is one of the most popular techniques in this area, using both unsupervised and supervised learning [10].…”