Semantic Clustering and Convolutional Neural Network for Short Text Categorization

Abstract

Short texts usually encounter data sparsity and ambiguity problems in representations for their lack of context. In this paper, we propose a novel method to model short texts based on semantic clustering and convolutional neural network. Particularly, we first discover semantic cliques in embedding spaces by a fast clustering algorithm. Then, multi-scale semantic units are detected under the supervision of semantic cliques, which introduce useful external knowledge for short texts. These meaningful semantic units are combined and fed into convolutional layer, followed by max-pooling operation. Experimental results on two open benchmarks validate the effectiveness of the proposed method.

Are Deep Learning Approaches Suitable for Natural Language Processing?

“…Another application uses Recurrent Neural Networks for question answering systems about paragraphs [15], and a Neural Responding Machine (NRM) has been proposed for a Short-Text Conversation generator, which is based on neural networks [21]. In addition, Recurrent Neural Networks models offered state-of-the-art performance for sentiment classification [13], target-dependent sentiment classification [25] and question answering [15]. Adaptive Recurrent Neural Network (AdaRNN) is introduced for sentiment classification in Twitter promotions based on the context and syntactic relationships between words [2].…”

Text mining to detect indications of fraud in annual reports worldwide

“…The research discussed in this paper applied a simple CNN architecture that demonstrated successful results in various domains (Kim, 2014). Many variants on this architecture were proposed (Johnson and Zhang, 2014;Chen et al, 2015;Ma et al, 2015;Nguyen and Grishman, 2015;Wang et al, 2015;Chen and Ku, 2016;Laha and Raykar, 2016). It is conceivable that one of these variants may achieve better results for detecting indications of fraud in texts.…”

“…Both Ma et al (2015) and Zhang et al (2016b) reported an accuracy of 95.6%. Wang et al (2015) achieved the highest accuracy, 97.2%, with GloVe word embeddings.…”

“…The 29 layer model of Conneau et al (2016) improves the results; however, only for two out of the four data sets, the CNN performs better than the bag-of-words model. Wang et al (2015) developed a CNN with one convolutional layer and a layer that extracts several representations of the texts by applying multiple windows with various width over the pre-trained word embeddings. They experimented with three types of word embeddings, Senna trained on 130k Wikipedia articles, GloVe trained on 400k Wikipedia articles and Word2Vec trained on 3 million Google News messages.…”

“…The previously discussed models of Kim (2014), Kalchbrenner et al (2014), Ma et al (2015), Wang et al (2015) and Zhang et al (2016b) were applied to the TREC questions data set that includes six question types: abbreviation, entity, description, human, location and numeric. The model of Kalchbrenner et al (2014) achieved an accuracy of 93%.…”