Rethinking Complex Neural Network Architectures for Document Classification

Abstract

Neural network models for many NLP tasks have grown increasingly complex in recent years, making training and deployment more difficult. A number of recent papers have questioned the necessity of such architectures and found that well-executed, simpler models are quite effective. We show that this is also the case for document classification: in a large-scale reproducibility study of several recent neural models, we find that a simple BiLSTM architecture with appropriate regularization yields accuracy and F 1 that are either competitive or exceed the state of the art on four standard benchmark datasets. Surprisingly, our simple model is able to achieve these results without attention mechanisms. While these regularization techniques, borrowed from language modeling, are not novel, to our knowledge we are the first to apply them in this context. Our work provides an opensource platform and the foundation for future work in document classification.

Improving Document-Level Sentiment Classification Using Importance of Sentences

“…For our experiments, we used the IMDB dataset (135,669 documents) [ 28 ], the Yelp-hotel dataset (34,961 documents) [ 29 ], the Yelp-rest dataset (178,239 documents) [ 29 ], and the Amazon dataset (83,159 documents) [ 29 ]. The IMDB dataset is a movie review dataset annotated with 10-scale polarities.…”

“…In Table 4 , Kim-CNN [ 8 ] is a sentence classification model that shows good performances, although it uses simple CNNs. Adhikari-logistic regression [ 28 ] and Adhikari-support vector machine [ 28 ] are text classification models based on logistic regression and support vector machine, in which the term frequency and inversed document-frequency scores are used as features, respectively. HAN [ 32 ] extracts meaningful features by modeling the hierarchical structure of a document and classifies the document into predefined classes using two levels of attention mechanisms: Word-level attentions and sentence-level attentions.…”

“…HAN [ 32 ] extracts meaningful features by modeling the hierarchical structure of a document and classifies the document into predefined classes using two levels of attention mechanisms: Word-level attentions and sentence-level attentions. LSTM-Reg [ 28 ] is a sentiment classification model based on single-layer bidirectional long short-term memory (BiLSTM). Knowledge distillation (KD)-LSTM [ 28 ] is a modified version of LSTM that uses the KD scheme to increase performances using fine-tuned BERT-Large [ 16 ].…”

“…To statistically validate the performance differences, we performed t -tests between the proposed model and the comparison models using the accuracies as the input values of the t -test. The p -values were 0.000311 between the proposed model and Kim-CNN [ 8 ], 0.000164 between the proposed model and HAN [ 32 ], 0.000164 between the proposed model and ALBERT-Base, and 0.001341 between the proposed model and LSTM-Reg [ 28 ], respectively. This implies that the performances are statistically meaningful at a significance level of 99%.…”

Screenplay Quality Assessment: Can We Predict Who Gets Nominated?

“…By nature, a movie should be tough to be cleanly categorized, due to its length, complex storyline and turns, and the lack of evaluative criteria. Prior works in document classification (Yang et al, 2016;Liu et al, 2017;Adhikari et al, 2019;Johnson and Zhang, 2015) evaluated on datasets with small document size (Reuters, IMDB, Yelp, etc.). However, our document size on average is at least 65 times longer, which may be challenging for NN-based models to train due to long sequences and the associated computational burden.…”