Recognizing text genres with simple metrics using discriminant analysis

Abstract

A simple method for categorizing texts into pre-determined text genre categories using the statistical standard technique of discriminant analysis is demonstrated with application to the Brown corpus. Discriminant analysis makes it possible use a large number of parameters that may be specific for a certain corpus or information stream, and combine them into a small number of functions, with the parameters weighted on basis of how useful they are for discriminating text genres. An application to information retrieval is discussed. Text TypesThere are different types of text. Texts "about" the same thing may be in differing genres, of different types, and of varying quality. Texts vary along several parameters, all relevant for the general information retrieval problem of matching reader needs and texts. Given this variation, in a text retrieval context the problems are (i) identifying genres, and (ii) choosing criteria to cluster texts of the same genre, with predictable precision and recall. This should not be confused with the issue of identifying topics, and choosing criteria that discriminate one topic from another. Although not orthogonal to genre-dependent variation, the variation that relates directly to content and topic is along other dimensions. Naturally, there is co-variance. Texts about certain topics may only occur in certain genres, and texts in certain genres may only treat certain topics; most topics do, however, occur in several genres, which is what interests us here. Douglas Biber has studied text variation along several parameters, and found that texts can be considered to vary along five dimensions. In his study, he clusters features according to covariance, to find underlying dimensions (1989). We wish to find a method for identifying easily computable parameters that rapidly classify previously unseen texts in general classes and along a small set -smaller than Biber's five -of dimensions, such that they can be explained in intuitively simple terms to the user of an information retrieval application. Our aim is to take a set of texts that has been selected by some sort of crude semantic analysis such as is typically performed by an information retrieval system and partition it further by genre or text type, and to display this variation as simply as possible in one or two dimensions. MethodWe start by using features similar to those first investigated by Biber, but we concentrate on those that are easy to compute assuming we have a part of speech tagger Church, 1988), such as such as third person pronoun occurrence rate as opposed to 'general hedges ' (Biber, 1989). More and more of Biber's features will be available with the advent of more proficient analysis programs, for instance if complete surface syntactic parsing were performed before categorization (Voutilainen & Tapanainen, 1993).We then use discriminant analysis, a technique from descriptive statistics. Discriminant analysis takes a set of precategorized individuals and data on their variation on a number of parameters, ...

Computational methods in authorship attribution

“…Many studies since that of Mosteller and Wallace have shown the efficacy of function words for authorship attribution in different scenarios (Morton 1978;Burrows 1987;Karlgren & Cutting 1994;Merriam & Matthews 1994;Kessler et al 1997;Argamon et al 1998;Holmes 1998;de Vel et al 2001;Holmes et al 2001aHolmes et al , 2001bBaayen et al 2002;Binongo 2003;Juola & Baayen 2003;Zhao & Zobel 2005;Argamon & Levitan 2005;Koppel et al 2005Koppel et al , 2006a, confirming the hypothesis that different authors tend to have different characteristic patterns of function word use.…”

A Golden Resource for Named Entity Recognition in Portuguese

“…We wanted to investigate whether the sort of NEs in a given text might be a discriminating feature for classifying genre [20]. Preliminary results on this subject have already been published in [1].…”

Learning to classify documents according to genre

“…Karlgren [4,6,7] has done several experiments in genre classification. In [4] he shows that the texts that were judged relevant to a set of TREC queries differ systematically (in terms of style) from the texts that were not relevant.…”