Augmenting Naive Bayes Classifiers with Statistical Language Models

Abstract

We augment naive Bayes models with statistical n-gram language models to address shortcomings of the standard naive Bayes text classifier. The result is a generalized naive Bayes * Most research was conducted while the authors were at the School of Computer Science at University of Waterloo, Canada. 1classifier which allows for a local Markov dependence among observations; a model we refer to as the Chain Augmented Naive Bayes (CAN) Bayes classifier. CAN models have two advantages over standard naive Bayes classifiers. First, they relax some of the independence assumptions of naive Bayes-allowing a local Markov chain dependence in the observed variables-while still permitting efficient inference and learning. Second, they permit straightforward application of sophisticated smoothing techniques from statistical language modeling, which allows one to obtain better parameter estimates than the standard Laplace smoothing used in naive Bayes classification. In this paper, we introduce CAN models and apply them to various text classification problems. To demonstrate the language independent and task independent nature of these classifiers, we present experimental results on several text classification problems-authorship attribution, text genre classification, and topic detection-in several languages-Greek, English, Japanese and Chinese. We then systematically study the key factors in the CAN model that can influence the classification performance, and analyze the strengths and weaknesses of the model.

Computational methods in authorship attribution

“…More recently, Graham et al (2005) and Zheng et al (2006) used neural networks on a wide variety of features. Other studies used k-nearest neighbor (Kjell et al 1995;Hoorn et al 1999;Zhao & Zobel 2005), Naive Bayes (Kjell 1994a;Hoorn et al 1999;Peng et al 2004), rule learners (Holmes & Forsyth 1995;Holmes 1998;Argamon et al 1998;Koppel & Schler 2003;Abbasi & Chen 2005;Zheng et al 2006), support vector machines (De Vel et al 2001;Diederich et al 2003;Koppel & Schler 2003, Abbasi & Chen 2005Koppel et al 2005;Zheng et al 2006), Winnow (Koppel et al 2002;Argamon et al 2003;Koppel et al 2006a), and Bayesian regression Madigan et al 2006;Argamon et al 2008). Further details regarding these studies can be found in the Appendix.…”

Tensor Space Models for Authorship Identification

“…An alternative way to automatically define the function word set is to extract the most frequent words in a corpus [24,29]. There are also attempts to use word n-grams to exploit contextual information [27,7]. However, this process considerably increases the dimensionality of the problem and has not produced encouraging results so far.…”

“…Such powerful machine learning algorithms can effectively cope with high dimensional and sparse data. Another approach is to apply a generative model, like a naïve Bayes model [27]. Yet another approach is to estimate the similarity between two texts [4,17].…”

“…The other main issue is the development of attribution methodologies to assign texts to one candidate author. So far, the proposed attribution models comprise standard discriminative algorithms (e.g., support vector machines) [9] and generative models (e.g., Bayesian methods) [27] as well as models specifically designed for authorship identification tasks [21,29].…”

A Language Modelling Approach to Linking Criminal Styles with Offender Characteristics

“…Researchers [2,14,15] have previously used language models for document classification and such an approach was essentially Bayesian. We too adopt a Bayesian approach but, in common with most IR applications, apply models that are unigram in that they consider each term independently and do not take account of the preceding tokens.…”