Text Categorization Based on Topic Model

Zhou, Shibin; Li, Kan; Liu, Yushu

doi:10.2991/ijcis.2009.2.4.8

Cited by 2 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Documents are represented as a mixture of topic distributions and topics as a mixture of words distributions. The representations are inferred from a large training corpus, and when used for text categorization, information about the text categories is not taken into account (e.g., [14,52,55,56]). Interestingly, Siefkes et al [45] perform spam filtering based on orthogonal sparse bigrams.…”

Section: Related Workmentioning

confidence: 99%

Highly discriminative statistical features for email classification

2011

View full text Add to dashboard Cite

This paper reports on email classification and filtering, more specifically on spam versus ham and phishing versus spam classification, based on content features. We test the validity of several novel statistical feature extraction methods. The methods rely on dimensionality reduction in order to retain the most informative and discriminative features. We successfully test our methods under two schemas. The first one is a classic classification scenario using a 10-fold cross-validation technique for several corpora, including four ground truth standard corpora: Ling-Spam, SpamAssassin, PU1, and a subset of the TREC 2007 spam corpus, and one proprietary corpus. In the second schema, we test the anticipatory properties of our extracted features and classification models with two proprietary datasets, formed by phishing and spam emails sorted by date, and with the public TREC 2007 spam corpus. The contributions of our work are an exhaustive comparison of several feature selection and extraction methods in the frame of email classification on different benchmarking corpora, and the evidence that especially the technique of biased discriminant analysis offers better discriminative features for the classification, gives stable classification results notwithstanding the amount of features chosen, and robustly retains their discriminative value over time and data setups. These findings are especially useful in a commercial setting, where short profile rules are built based on a limited number of features for filtering emails.

show abstract