A new term weighting scheme based on class specific document frequency for document representation and classification

Abstract

Document classification is usually more challenging than numerical data classification, because it is much more difficult to effectively represent documents than numerical data for classification purposes. Vector space model (VSM) has been widely used for document representation for classification, in which a document is represented by a vector of feature values based on a bag of words. This paper proposes a new feature for document representation under the VSM framework, class specific document frequency (CSDF), which leads to a novel term weighting scheme based on term frequency (TF), term presence (TP), and the newly proposed feature. The experimental results show that the proposed features, CSDF and TF-CSDF, effectively improve the performance of document classification in comparison with other widely used VSM document representations.

Hybrid approaches to feature subset selection for data classification in high-dimensional feature space

“…After pre-processing, the total number of words for the emails_v1 dataset, emails_v2 dataset, 20 newsgroups dataset, and routers dataset was 1,014, 465, 2,591, and 412 respectively. After that, four term-weighting schemes were applied to the words in the BOW: term frequency (TF), term presence (TP), term frequency and inverse document frequency (TF-IDF), [36] and term presence and class-specific document frequency (TP-CSDF), [37] to generate numerical features. The other three datasets are numerical.…”

Re-ranking Google search returned web documents using document classification scores

“…[14] In this paper, the effectiveness of CSDF will be further investigated for web document representation for classification purposes. The basic idea of CSDF is that a term in a document is very important for classifying documents if it is more frequent inside the document and other documents belonging to the same class as well but less frequent in documents belonging to different classes.…”

“…For document representation, the class specific document frequency (CSDF) weighting method is adopted, which has been demonstrated to effectively improve the performance of document classification in comparison with other widely used vector space model (VSM) based document representations. [14] This paper proposes a new ranking method called GCrank that combines the original Google ranking scores and the LDA classification scores of the Google search returned web documents to improve ranking performance, which is demonstrated by experimental results in terms of several widely used ranking performance criteria.…”