Hybrid Neural-global Minimization Method of Logical Rule Extraction

Abstract

Methodology of extraction of optimal sets of logical rules using neural networks and global minimization procedures has been developed. Initial rules are extracted using density estimation neural networks with rectangular functions or multi-layered perceptron (MLP) networks trained with constrained backpropagation algorithm, transforming MLPs into simpler networks performing logical functions. A constructive algorithm called C-MLP2LN is proposed, in which rules of increasing specificity are generated consecutively by adding more nodes to the network. Neural rule extraction is followed by optimization of rules using global minimization techniques. Estimation of confidence of various sets of rules is discussed. The hybrid approach to rule extraction has been applied to a number of benchmark and real life problems with very good results.Keywords: computational intelligence, neural networks, extraction of logical rules, data mining Logical rules -introduction.Why should one use logical rules if other methods of classification -machine learning, pattern recognition or neural networks -may be easier to use and give better results? Adaptive systems M W , including the most popular neural network models, are useful classifiers that adjust internal parameters W performing vector mappings from the input to the output space. Although they may achieve high accuracy of classification the knowledge acquired by such systems is represented in a set of numerical parameters and architectures of networks in an incomprehensible way. Logical rules should be preferred over other methods of classification provided that the set of rules is not too complex and classification accuracy is sufficiently high. Surprisingly, in many applications simple rules proved to be more accurate and were able to generalize better than various machine and neural learning algorithms. Many statistical, pattern recognition and machine learning [1] methods of finding logical rules have been designed in the past. Neural networks are also used to extract logical rules and select classification features. Unfortunately systematic comparison of neural and machine learning methods is still missing. Many neural rule extraction methods have recently been reviewed and compared experimentally [2], therefore we will not discuss them here. Neural methods focus on analysis of parameters (weights and biases) of trained networks, trying to achieve high fidelity of performance, i.e. similar results of classification by extracted logical rules and by the original networks. Non-standard form of rules, such as M -of-N rules (M out of N antecedents should be true), fuzzy rules, or decision trees [1] are sometimes useful but in this paper we will consider only standard IF ... THEN propositional rules.In classification problems propositional rules may take several forms. Very general form of such rule is: IF X ∈ K (i) THEN Class(X) = C i , i.e. if X belongs to the cluster K (i) then its class is C i =Class(K (i) ), the same as for all vectors in this cluster. If clusters ...

A new methodology of extraction, optimization and application of crisp and fuzzy logical rules

“…In many applications simple crisp logical rules proved to be more accurate and were able to generalize better than many machine and neural learning algorithms [5]. In other applications fuzzification of logical rules gave more accurate results [6].…”

Optimization of logical rules derived by neural procedures

“…N EURAL methodology of crisp logical rule extraction developed by our group has been described in a series of papers [1], [2], [4], [10], [13], therefore only a very brief summary is given here.…”

“…Recently we have presented a complete methodology for extraction, optimization and application of logical rules [2], [4]. The last two steps are largely neglected in the literature, with current emphasis being still on the extraction methods.…”

Neural networks in non-Euclidean metric spaces

“…Two classes, Iris virginica and Iris versicolor, overlap, and therefore a perfect partition of the input space into separate classes is not possible. An optimal solution (from the point of view of generalization) contains 3 errors [18] and may be obtained using only two of the four input features (x 3 and x 4 ), therefore results are easy to display and only those two features have been left in simulations described below. The data has been standardized and rescaled to fit inside a square with ±1 corners.…”