Deep Unordered Composition Rivals Syntactic Methods for Text Classification

Abstract

Many existing deep learning models for natural language processing tasks focus on learning the compositionality of their inputs, which requires many expensive computations. We present a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time. While our model is syntactically-ignorant, we show significant improvements over previous bag-of-words models by deepening our network and applying a novel variant of dropout. Moreover, our model performs better than syntactic models on datasets with high syntactic variance. We show that our model makes similar errors to syntactically-aware models, indicating that for the tasks we consider, nonlinearly transforming the input is more important than tailoring a network to incorporate word order and syntax.

Guiding the Training of Distributed Text Representation with Supervised Weighting Scheme for Sentiment Analysis

“…Upon word order, Four models in 'complex' group use the combination of neural networks to capture more complex information from textual data. Compared with neural models such as CNNs and RNNs, BOW models have the advantages of being efficient and robust, but are inferior to those complex models in accuracies [7]. NBSVM introduces supervised weighting schemes into bag-of-ngrams representation and provides strong baselines.…”

“…Instead of constructing complex compositions upon word embeddings, these models basically ignore order and syntax information. Representative neural bagof-words models include Deep Averaging Network (DAN) [7] and Paragraph Vector (PV) [11]. DAN firstly takes the average of word embeddings as the inputs and then constructs multiple neural layers upon them.…”

Adversarial Training for Relation Extraction

“…There are also other works for regularizing classifiers by adding random noise to the data, such as dropout (Srivastava et al, 2014) and its variant for NLP tasks, word dropout (Iyyer et al, 2015). Xie et al (2017) discusses various data noising techniques for language models.…”

Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts

“…This feature network takes as input the average value of pretrained embeddings 3 for the tokens in the character description (we remove stopwords 4 ). This initial vector is passed through hidden layers to yield the feature embedding e (reminiscent of deep averaging networks by Iyyer et al (2015)). …”