Bi-Directional Recurrent Attentional Topic Model

Abstract

In a document, the topic distribution of a sentence depends on both the topics of its neighbored sentences and its own content, and it is usually affected by the topics of the neighbored sentences with different weights. The neighbored sentences of a sentence include the preceding sentences and the subsequent sentences. Meanwhile, it is natural that a document can be treated as a sequence of sentences. Most existing works for Bayesian document modeling do not take these points into consideration. To fill this gap, we propose a bi-Directional Recurrent Attentional Topic Model (bi-RATM) for document embedding. The bi-RATM not only takes advantage of the sequential orders among sentences but also uses the attention mechanism to model the relations among successive sentences. To support to the bi-RATM, we propose a bi-Directional Recurrent Attentional Bayesian Process (bi-RABP) to handle the sequences. Based on the bi-RABP, bi-RATM fully utilizes the bidirectional sequential information of the sentences in a document. Online bi-RATM is proposed to handle large-scale corpus. Experiments on two corpora show that the proposed model outperforms state-of-the-art methods on document modeling and classification.

bi-directional Bayesian probabilistic model based hybrid grained semantic matchmaking for Web service discovery

“…Specifically, web services are described by a specific semantic tagged language, e.g., SAWSDL (Semantic Annotations for Web services description language) and OWL-S (Web Ontology Language for Services). The approaches based on a topic model are another kind of semantics-aware service discovery approaches, and many related approaches have been proposed in recent years [3,17,18]. LDA-SVM [22] is proposed to handle the issue of labeling a large number of services when a service classifier is trained.…”

“…Different from the conventional methods on service discovery or clustering with topic modeling, the bi-SWTM assumes that the topic distribution of each sentence in service descriptions is not only determined by the concepts of the involved words but is also influenced by its preceding and subsequent sentences. This is the basic assumption in many text mining tasks, such as topic modeling [17] and neural language modeling [10]. In traditional topic modeling, the topics of a text follow a special distribution a prior such as a Dirichlet [3] or Normal [2] distributions.…”

“…This type of approach is designed to build the topic space learned by texts of services and aim to match the services with their latent topic features. With the development of topic models, the semantics are captured from various texts, such as sentences [16,17]. The semantics among the words and the paragraphs are highly interpretable and have been proven to be effective in many works in the area of information retrieval and text modeling [16,19].…”

“…With the development of topic models, the semantics are captured from various texts, such as sentences [16,17]. The semantics among the words and the paragraphs are highly interpretable and have been proven to be effective in many works in the area of information retrieval and text modeling [16,19].…”

Extracting Topics with Simultaneous Word Co-occurrence and Semantic Correlation Graphs: Neural Topic Modeling for Short Texts

“…Nowadays, the representatives include Neural Variational Document Model (NVDM) (Miao et al, 2016), Product of expert LDA (ProdLDA) (Srivastava and Sutton, 2017), and Embedded Topic Model (ETM) (Dieng et al, 2020), etc. Besides these "naive" neural variants of LDA, many other models have been investigated by applying (1) various neural modules to the topic encoder, e.g., recurrent module (Rezaee and Ferraro, 2020), attention mechanism (Li et al, 2020b), and graphical connection (Zhu et al, 2018;Yang et al, 2020), and (2) new learning paradigms, e.g., adversarial training (Wang et al, 2019), reinforcement learning (Gui et al, 2019), and lifelong learning (Gupta et al, 2020). However, despite their effectiveness on normal long texts, those models suffer from the sparsity problem of short texts (Zeng et al, 2018).…”