Bag-of-concepts: Comprehending document representation through clustering words in distributed representation

Abstract

Two document representation methods are mainly used in solving text mining problems. Known for its intuitive and simple interpretability, the bag-ofwords method represents a document vector by its word frequencies. However, this method suffers from the curse of dimensionality, and fails to preserve accurate proximity information when the number of unique words increases. Furthermore, this method assumes every word to be independent, disregarding the impact of semantically similar words on preserving document proximity. On the other hand, doc2vec, a basic neural network model, creates low dimensional vectors that successfully preserve the proximity information. However, it loses the interpretability as meanings behind each feature is indescribable. This paper proposes the bag-of-concepts method as an alternative document representation method that overcomes the weaknesses of these two methods. This proposed method creates concepts through clustering word vectors generated from word2vec, and uses the frequencies of these concept clusters to represent document vectors. Through these data-driven concepts, the proposed method incorporates the impact of semantically similar ii words on preserving document proximity effectively. With appropriate weighting scheme such as concept frequency-inverse document frequency, the proposed method provides better document representation than previously suggested methods, and also offers intuitive interpretability behind the generated document vectors. Based on the proposed method, subsequently constructed text mining models, such as decision tree, can also provide interpretable and intuitive reasons on why certain collections of documents are different from others.

Testing word embeddings for Polish

“…finding synonyms (Griffiths, Steyvers, & Tenenbaum, 2007), word clustering (Kovatchev, Salamo, & Marti, 2016), sentiment analysis (Duyu, Wei, Yang, Ming, Ting, & Bing, 2014), word sense disambiguation (Basile, Caputo, & Semeraro, 2014), metaphor recognition (Shutova, Sun, Gutierrez, Lichtenstein, & Narayanan, 2017), sentence paraphrasing , and documents classification (Kim, Kim, & Cho, 2017). Many problems are best solved with the help of DS models, e.g.…”

Evaluación de distintas técnicas de representación de texto y medidas de distancia de texto usando KNN para clasificación de documentos

“…The bag of words is the data representation technique used in most of the consulted literature [13,6,14,15,1,16]. It consists on representing each text document as a vector of frequencies [13].…”

“…Recent studies have evaluated text classification techniques mostly focused on classification algorithms [4] and distance measures [3]. There have also been works related to the comparison of preprocessing methods and document representations [15,14]. This work intends to achieve a similar evaluation but combining the representations with the distance metrics.…”

Exchange-Based Diffusion in Hb-Graphs: Highlighting Complex Relationships

“…Bags of concepts are an extension of bags of words to successive concepts in a text [38]. A recent extension of these concepts is given in [39] where bag of graphs are introduced to encode in graphs the local structure of a digital object: bags of graphs are declined into bags of singleton graphs and bags of visual graphs.…”