Estimating Topic Modeling Performance with Sharma–Mittal Entropy

Koltcov, Sergei; Ignatenko, Vera; Koltsova, Olessia

doi:10.3390/e21070660

Cited by 29 publications

(29 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, topic model is described by two observable parameters: (1) the sum of probabilities of highly probable words); (2) the number of highly probable words, N. Therefore, partition function (statistical sum) of a topic model can be expressed as Z q = ρ • (qP) q , where q = 1/T [34]. Correspondingly, Renyi entropy of a topic model is expressed in terms of partition function as…”

Section: Entropy Approach For Analysis Of Topic Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing the Influence of Hyper-parameters and Regularizers of Topic Modeling in Terms of Renyi Entropy

Koltcov

Ignatenko

Boukhers

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

Topic modeling is a popular technique for clustering large collections of text documents. A variety of different types of regularization is implemented in topic modeling. In this paper, we propose a novel approach for analyzing the influence of different regularization types on results of topic modeling. Based on Renyi entropy, this approach is inspired by the concepts from statistical physics, where an inferred topical structure of a collection can be considered an information statistical system residing in a non-equilibrium state. By testing our approach on four models—Probabilistic Latent Semantic Analysis (pLSA), Additive Regularization of Topic Models (BigARTM), Latent Dirichlet Allocation (LDA) with Gibbs sampling, LDA with variational inference (VLDA)—we, first of all, show that the minimum of Renyi entropy coincides with the “true” number of topics, as determined in two labelled collections. Simultaneously, we find that Hierarchical Dirichlet Process (HDP) model as a well-known approach for topic number optimization fails to detect such optimum. Next, we demonstrate that large values of the regularization coefficient in BigARTM significantly shift the minimum of entropy from the topic number optimum, which effect is not observed for hyper-parameters in LDA with Gibbs sampling. We conclude that regularization may introduce unpredictable distortions into topic models that need further research.

show abstract

Section: Entropy Approach For Analysis Of Topic Modelsmentioning

confidence: 99%

“…A more detailed explanation of formulating Renyi entropy for topic models can be found in [7,34]. Application of Renyi entropy for investigation of TM results is useful due to the following reasons.…”

Section: Entropy Approach For Analysis Of Topic Modelsmentioning

confidence: 99%

Analyzing the Influence of Hyper-parameters and Regularizers of Topic Modeling in Terms of Renyi Entropy

Koltcov

Ignatenko

Boukhers

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…The entropy approach to TM tuning is based primarily on computing Rényi entropy for each topic solution while varying the number of topics and hyperparameters [4,8]. For TM, the Rényi entropy is expressed as follows:…”

Section: Entropic Approach For Determining the Optimal Number Of Topicsmentioning

confidence: 99%

“…In this area, S R q reaches its minimum. It has been shown that the minimum Rényi entropy corresponds to the number of topics identified by human coders [8]. Hence, the search for the S R q minimum could, at least partly, substitute the manual labor of marking up document collections, substantially simplifying TM tuning on uncoded datasets.…”

Section: Entropic Approach For Determining the Optimal Number Of Topicsmentioning

confidence: 99%

Fast Tuning of Topic Models: An Application of Rényi Entropy and Renormalization Theory

Koltcov

Ignatenko

Pashakhin

2019

The 5th International Electronic Conference on Entropy and Its Applications

Self Cite

View full text Add to dashboard Cite

In practice, the critical step in building machine learning models of big data (BD) is costly in terms of time and the computing resources procedure of parameter tuning with a grid search. Due to the size, BD are comparable to mesoscopic physical systems. Hence, methods of statistical physics could be applied to BD. The paper shows that topic modeling demonstrates self-similar behavior under the condition of a varying number of clusters. Such behavior allows using a renormalization technique. The combination of a renormalization procedure with the Rényi entropy approach allows for fast searching of the optimal number of clusters. In this paper, the renormalization procedure is developed for the Latent Dirichlet Allocation (LDA) model with a variational Expectation-Maximization algorithm. The experiments were conducted on two document collections with a known number of clusters in two languages. The paper presents results for three versions of the renormalization procedure: (1) a renormalization with the random merging of clusters, (2) a renormalization based on minimal values of Kullback-Leibler divergence and (3) a renormalization with merging clusters with minimal values of Rényi entropy. The paper shows that the renormalization procedure allows finding the optimal number of topics 26 times faster than grid search without significant loss of quality.

show abstract

“…Thus, when conducting sentiment analysis of such texts, it is necessary to analyze not only the themes, but also the viewpoints and positions, that is, text orientation. Moreover, in terms of the dominant research methods in sentiment analysis, the situations of "Synonymy" and "Polysemy" in natural language and the semantic correlations between vocabulary and document are often ignored in the process of modeling [4,5]. Even more, other features, such as Semantic Structure, the latent semantic information of documents, are also ignored [6].…”

Section: Introductionmentioning

confidence: 99%

Application of Support Vector Machine (SVM) in the Sentiment Analysis of Twitter DataSet

et al. 2020

View full text Add to dashboard Cite

At present, in the mainstream sentiment analysis methods represented by the Support Vector Machine, the vocabulary and the latent semantic information involved in the text are not well considered, and sentiment analysis of text is dependent overly on the statistics of sentiment words. Thus, a Fisher kernel function based on Probabilistic Latent Semantic Analysis is proposed in this paper for sentiment analysis by Support Vector Machine. The Fisher kernel function based on the model is derived from the Probabilistic Latent Semantic Analysis model. By means of this method, latent semantic information involving the probability characteristics can be used as the classification characteristics, along with the improvement of the effect of classification for support vector machine, and the problem of ignoring the latent semantic characteristics in text sentiment analysis can be addressed. The results show that the effect of the method proposed in this paper, compared with the comparison method, is obviously improved.

show abstract

Estimating Topic Modeling Performance with Sharma–Mittal Entropy

Cited by 29 publications

References 56 publications

Analyzing the Influence of Hyper-parameters and Regularizers of Topic Modeling in Terms of Renyi Entropy

Analyzing the Influence of Hyper-parameters and Regularizers of Topic Modeling in Terms of Renyi Entropy

Fast Tuning of Topic Models: An Application of Rényi Entropy and Renormalization Theory

Application of Support Vector Machine (SVM) in the Sentiment Analysis of Twitter DataSet

Contact Info

Product

Resources

About