An Information-Theoretic Approach to the Pre-Pruning of Classification Rules

Abstract

Abstract:The automatic induction of classification rules from examples is an important technique used in data mining. One of the problems encountered is the overfitting of rules to training data. In some cases this can lead to an excessively large number of rules, many of which have very little predictive value for unseen data. This paper is concerned with the reduction of overfitting. It introduces a technique known as J-pruning, based on the Jmeasure, an information theoretic means of quantifying the information content of a rule and applies this to two rule induction methods: one where the rules are generated via the intermediate representation of a decision tree and one where rules are generated directly from examples.

A Scalable Expressive Ensemble Learning Using Random Prism: A MapReduce Approach

“…In this paper, we adopt PrismTCS which is a computationally efficient member of the Prism family, and also maintains a similar predictive accuracy compared with the original Prism classifier [5]. A good computational efficiency is needed as ensemble learners generally do not scale well to large datasets.…”

“…However, R-PrismTCS classifiers could also be parallelised. The Parallel Modular Classification Rule Induction (PMCRI) framework [19] for parallelising, amongst others, the PrismTCS [5] classifier, can be used for parallelising the R-PrismTCS classifier also. This is due to the similarity of the R-PrismTCS and PrismTCS classifiers.…”

“…'Divide and conquer' based classifiers produce a decision tree, such as Quinlan's C4.5 decision tree induction algorithm [18]; 'separate and conquer' based classifiers produce a set of IF...THEN classification rules, such as the Prism family of algorithms [8,4,5]. As pointed out in [20], most ensemble classifiers are based on the 'divide and conquer' approach even though Prism classifiers have shown to be less vulnerable to overfitting compared with decision tree based classifiers [4].…”

eRules: A Modular Adaptive Classification Rule Learning Algorithm for Data Streams

“…Even so Prism has been shown to be less vulnerable to overfitting compared with decision trees, it is not immune. Hence pruning methods for Prism have been developed such as J-pruning [6] in order to make Prism generalising better on the input data. Because of J-pruning's generalisation capabilities we use Prism with J-pruning for the induction of rules in eRules hence it is briefly described here.…”

“…A possible candidate for that could be the PrismTCS classifier [6] which uses a default rule for unclassified instances.…”