Neural Topic Modeling with Bidirectional Adversarial Training

Abstract

Recent years have witnessed a surge of interests of using neural topic models for automatic topic extraction from text, since they avoid the complicated mathematical derivations for model inference as in traditional topic models such as Latent Dirichlet Allocation (LDA). However, these models either typically assume improper prior (e.g. Gaussian or Logistic Normal) over latent topic space or could not infer topic distribution for a given document. To address these limitations, we propose a neural topic modeling approach, called Bidirectional Adversarial Topic (BAT) model, which represents the first attempt of applying bidirectional adversarial training for neural topic modeling. The proposed BAT builds a twoway projection between the document-topic distribution and the document-word distribution. It uses a generator to capture the semantic patterns from texts and an encoder for topic inference. Furthermore, to incorporate word relatedness information, the Bidirectional Adversarial Topic model with Gaussian (Gaussian-BAT) is extended from BAT. To verify the effectiveness of BAT and Gaussian-BAT, three benchmark corpora are used in our experiments. The experimental results show that BAT and Gaussian-BAT obtain more coherent topics, outperforming several competitive baselines. Moreover, when performing text clustering based on the extracted topics, our models outperform all the baselines, with more significant improvements achieved by Gaussian-BAT where an increase of near 6% is observed in accuracy.

Self-Supervised Neural Topic Modeling

“…As such since one important goal of topic models is to be used as tools for humans to make sense of and explore document collections, the recent trend is to evaluate topics based on coherence metrics. The work of (Röder et al, 2015) and (Wang et al, 2020) are examples of topic coherence evaluation. Here, we take the same approach to evaluation.…”

“…Another model based on bi-directional GANs is the Gaussian-BAT (Wang et al, 2020) which uses a Drichlet prior and can infer topic distributions of input documents. Additionally, Gaussian-BAT models a topic using a multivariate Gaussian and incorporates the word relatedness into the modeling process.…”

“…Subsequently, several neural topic models emerged. However, some of these models came with limitations such as: (1) not being able to compute per-document topic distributions, e.g., NVDM (Miao et al, 2016) , (2) difficulty of training with respect to computational cost, e.g., topic models based on Generative Adversarial Networks (GANs) (Wang et al, 2020). ( 3) Priors based on distributions such as a logistic normal distribution (Miao et al, 2016) which may not always model the cooccurrence behaviour among words realistically, as they expect topic proportions of a corpus to follow such patterns.…”

Neural labeled LDA: a topic model for semi-supervised document classification

“…To overcome the problem that the Dirichlet distribution is not a location scale family, which hinders the reparameterization utilized in the VAE framework, Srivastava and Sutton (2017) employ a Laplace approximation for modeling a Dirichlet prior of the latent variables; Joo et al (2020) approximate the inverse cumulative distribution function of the Gamma distribution, which is a component of the Dirichlet distribution; Zhang et al (2018) utilize the Weibull distribution; and Burkhardt and Kramer (2019a) solve this problem based on rejection samples. Meanwhile, to utilize Dirichlet prior, Wang et al (2020a) abandon the VAE framework, and propose Bidirectional Adversarial Topic (BAT) model, which applies bidirectional adversarial training for neural topic modeling. All these methods of unsupervised neural topic models have achieved competitive results.…”

Neural Topic Modeling with Cycle-Consistent Adversarial Training

“…The proposed ToMCAT is partly inspired by ATM but differs in its capability of inferring documentspecific topic distributions and incorporating supervision information. BAT (Wang et al, 2020) is an extension to ATM that employs bidirectional adversarial training (Donahue et al, 2016) for documentspecific topic distribution inference. Although BAT similarly utilizes an adversarial training objective to guide the learning of topic distribution, there are some major differences.…”