The Dynamic Embedded Topic Model

Dieng, Adji B.; Ruiz, Francisco J. R.; Blei, David M.

doi:10.48550/arxiv.1907.05545

Cited by 18 publications

(27 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, GRADE is also related to dynamic topic modelling (Dieng et al, 2019;Blei and Lafferty, 2006) as both can also be viewed as state-space models. The difference is that in GRADE we are dealing with multinomial distributions over the nodes and communities instead of topics and words.…”

Section: Related Workmentioning

confidence: 99%

“…The difference is that in GRADE we are dealing with multinomial distributions over the nodes and communities instead of topics and words. Moreover, some works like (Bamler and Mandt, 2017;Rudolph and Blei, 2018) have focused on the shift of word meaning over time, and others such as (Dieng et al, 2019) model the evolution of documents. In contrast, GRADE assumes both nodes and communities undergo temporal semantic shift.…”

Section: Related Workmentioning

confidence: 99%

“…In our work, we assume that the semantic meaning of communities and the proportions over the communities for each node evolve simultaneously over time. Following an approach introduced in dynamic topic modelling (Dieng et al, 2019), we encode temporal evolution in our method by assuming a random walk prior over the representations between time steps. Furthermore, we draw inspiration from social networks where a user's preferences can shift from one community to another.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

GRADE: Graph Dynamic Embedding

Spasov¹,

Stefano²,

Lió³

et al. 2020

Preprint

View full text Add to dashboard Cite

Representation learning of static and more recently dynamically evolving graphs has gained noticeable attention. Existing approaches for modelling graph dynamics focus extensively on the evolution of individual nodes independently of the evolution of mesoscale community structures. As a result, current methods do not provide useful tools to study and cannot explicitly capture temporal community dynamics. To address this challenge, we propose GRADE -a probabilistic model that learns to generate evolving node and community representations by imposing a random walk prior over their trajectories. Our model also learns node community membership which is updated between time steps via a transition matrix. At each time step link generation is performed by first assigning node membership from a distribution over the communities, and then sampling a neighbor from a distribution over the nodes for the assigned community. We parametrize the node and community distributions with neural networks and learn their parameters via variational inference. Experiments demonstrate GRADE meets or outperforms baselines in dynamic link prediction, shows favourable performance on dynamic community detection, and identifies coherent and interpretable evolving communities.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GRADE: Graph Dynamic Embedding

Spasov¹,

Stefano²,

Lió³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, a lot of work is harnessing topic modeling (Blei et al 2003) along with word vectors to learn better word and sentence representations, e.g., LDA (Chen and Liu 2014), weight-BoC (Kim, Kim, and Cho 2017), TWE , NTSG (Liu, Qiu, and Huang 2015), WTM (Fu et al 2016), w2v-LDA (Nguyen et al 2015, TV+MeanWV (Li et al 2016a), LTSG (Law et al 2017), Gaussian-LDA (Das, Zaheer, and Dyer 2015), Topic2Vec (Niu et al 2015), TM (Dieng, Ruiz, and Blei 2019b), LDA2vec (Moody 2016), D-ETM (Dieng, Ruiz, and Blei 2019a) and MvTM . (Kiros et al 2015) propose skip-thought document embedding vectors which transformed the idea of abstracting the distributional hypothesis from word to sentence level.…”

Section: Related Workmentioning

confidence: 99%

P-SIF: Document Embeddings Using Partition Averaging

Gupta¹,

Saw²,

Nokhiz³

et al. 2020

Preprint

View full text Add to dashboard Cite

Simple weighted averaging of word vectors often yields effective representations for sentences which outperform sophisticated seq2seq neural models in many tasks. While it is desirable to use the same method to represent documents as well, unfortunately, the effectiveness is lost when representing long documents involving multiple sentences. One of the key reasons is that a longer document is likely to contain words from many different topics; hence, creating a single vector while ignoring all the topical structure is unlikely to yield an effective document representation. This problem is less acute in single sentences and other short text fragments where the presence of a single topic is most likely. To alleviate this problem, we present P-SIF, a partitioned word averaging model to represent long documents. P-SIF retains the simplicity of simple weighted word averaging while taking a document's topical structure into account. In particular, P-SIF learns topic-specific vectors from a document and finally concatenates them all to represent the overall document. We provide theoretical justifications on the correctness of P-SIF. Through a comprehensive set of experiments, we demonstrate P-SIF's effectiveness compared to simple weighted averaging and many other baselines.

show abstract

“…Then we develop the Sawtooth Connection technique to capture the dependencies between the topics at different layers, where the factor loading at layer l is the factor score at layer l − 1, which enables the hierarchical topics to be coupled together across all layers. Our work is inspired by both GBN (Zhou et al, 2015), a multi-stochastic-layer hierarchical topic model, and the embedding topic models (Dieng et al, 2019;2020), which represent the words and single layer topic as embedding vectors. The proposed Sawtooth Connector is a novel method that combines the advantages of both models for hierarchical topic modeling.…”

Section: Introductionmentioning

confidence: 99%

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Duan

Wang

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Hierarchical topic models such as the gamma belief network (GBN) have delivered promising results in mining multi-layer document representations and discovering interpretable topic taxonomies. However, they often assume in the prior that the topics at each layer are independently drawn from the Dirichlet distribution, ignoring the dependencies between the topics both at the same layer and across different layers. To relax this assumption, we propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space. Specifically, both the words and topics are represented as embedding vectors of the same dimension. The topic matrix at a layer is factorized into the product of a factor loading matrix and a topic embedding matrix, the transpose of which is set as the factor loading matrix of the layer above. Repeating this particular type of factorization, which shares components between adjacent layers, leads to a structure referred to as sawtooth factorization. An auto-encoding variational inference network is constructed to optimize the model parameter via stochastic gradient descent. Experiments on big corpora show that our models outperform other neural topic models on extracting deeper interpretable topics and deriving better document representations.

show abstract

The Dynamic Embedded Topic Model

Cited by 18 publications

References 25 publications

GRADE: Graph Dynamic Embedding

GRADE: Graph Dynamic Embedding

P-SIF: Document Embeddings Using Partition Averaging

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Contact Info

Product

Resources

About