CB3: An Adaptive Error Function for Backpropagation Training

Rimer, M.E.; Martinez, Tony

doi:10.1007/s11063-006-9014-9

Cited by 12 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In so doing, they perform relatively minimal updates to network parameters in order to discourage premature weight saturation and overfitting. CB3 is a CB approach that learns the error function to be used while training [110]. This is accomplished by learning pattern confidence margins during training, which are used to dynamically set output target values for each training pattern.…”

Section: Criterion Functionsmentioning

confidence: 99%

Fundamentals of Machine Learning

Swamy²

2013

Neural Networks and Statistical Learning

View full text Add to dashboard Cite

Learning is a fundamental capability of neural networks. Learning rules are algorithms for finding suitable weights W and/or other network parameters. Learning of a neural network can be viewed as a nonlinear optimization problem for finding a set of network parameters that minimize the cost function for given examples. This kind of parameter estimation is also called a learning or training algorithm.Neural networks are usually trained by epoch. An epoch is a complete run when all the training examples are presented to the network and are processed using the learning algorithm only once. After learning, a neural network represents a complex relationship, and possesses the ability for generalization. To control a learning process, a criterion is defined to decide the time for terminating the process. The complexity of an algorithm is usually denoted as O(m), indicating that the order of number of floating-point operations is m.Learning methods are conventionally divided into supervised, unsupervised, and reinforcement learning; these schemes are illustrated in Fig. 2.1. x p and y p are the input and output of the pth pattern in the training set,ŷ p is the neural network output for the pth input, and E is an error function. From a statistical viewpoint, unsupervised learning learns the pdf of the training set, p(x), while supervised learning learns about the pdf of p( y|x). Supervised learning is widely used in classification, approximation, control, modeling and identification, signal processing, and optimization. Unsupervised learning schemes are mainly used for clustering, vector quantization, feature extraction, signal coding, and data analysis. Reinforcement learning is usually used in control and artificial intelligence.In logic and statistical inference, transduction is reasoning from observed, specific (training) cases to specific (test) cases. In contrast, induction is reasoning from observed training cases to general rules, which are then applied to the test cases. Machine learning falls into two broad classes: inductive learning or transductive learning. Inductive learning pursues the standard goal in machine learning, which is to accurately classify the entire input space. In contrast, transductive learning focuses

show abstract

Section: Criterion Functionsmentioning

confidence: 99%

Fundamentals of Machine Learning

Swamy²

2013

Neural Networks and Statistical Learning

View full text Add to dashboard Cite

show abstract

“…In the meantime, researchers have identified that proper cost function is being an important factor to improve the performance of standard BP in terms of convergence speed [3]; [4]; [5]; [6]; [7]; [8]; [9]; [10]; [11]; [12]; [13]; [14], higher accuracy [15]; [16], and to overcome the problems of getting stuck into local minima [15]; [6]; [9]; [10]; [17]; [14].…”

Section: Introductionmentioning

confidence: 99%

“…It has been observed that, MSE cost function has some drawbacks such as incorrect saturation and tend to trap in local minima, resulting in slow convergence and poor performance [16]. MSE gives more emphasis on reducing the larger errors as compared to smaller errors due to the squaring that takes place.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Study of cost functions in Three Term Backpropagation for classification problems

Shamsuddin

Alwee

Kuppusamy

et al. 2009

2009 World Congress on Nature &Amp; Biologically Inspired Computing (NaBIC)

View full text Add to dashboard Cite

Three Term Backpropagation(BP) Network as proposed by Zweiri in 2003 has outperformed standard Two Term Backpropagation. However, further studies on Three Term Backpropagation in 2007 indicated that this network only surpassed standard BP for small scale datasets but not for medium and large scale datasets. It has also been observed that by using Mean Square Error (MSE) as a cost function in Three Term BP has some drawbacks, and these include incorrect saturation and tend to trap in local minima, resulting in slow convergence and poor performance. In this study, thorough experiments on implementing various cost functions are executed to probe the effectiveness of Three Term BP network. The cost functions under investigations include Mean Square Error (MSE), Bernoulli function, Modified cost function andImproved cost function. The results reveal that MSE is not an ideal cost function to be used for Three Term BP. Hence, alternative cost functions need to be considered when using BP network for classification problems.

show abstract

“…However, there is a large difference between making a parameter chaotic and chaos injection. Among other approaches, noise injection (NI) into the network [10], the addition of a penalty term in the cost function [1], a change of activation function [11] and a change of cost function [12] are worth mentioning. However, these approaches are greedy and they are only loosely motivated by biological counterparts.…”

mentioning

confidence: 99%

Injecting Chaos in Feedforward Neural Networks

Ahmed

Shahjahan

Murase

2011

Neural Process Lett

View full text Add to dashboard Cite

Chaos appears in many natural and artificial systems; accordingly, we propose a method that injects chaos into a supervised feed forward neural network (NN). The chaos is injected simultaneously in the learnable temperature coefficient of the sigmoid activation function and in the weights of the NN. This is functionally different from the idea of noise injection (NI) which is relatively distant from biological realism. We investigate whether chaos injection is more efficient than standard back propagation, adaptive neuron model, and NI algorithms by applying these techniques to different benchmark classification problems such as heart disease, glass, breast cancer, and diabetes identification, and time series prediction. In each case chaos injection is superior to the standard approaches in terms of generalization ability and convergence rate. The performance of the proposed method is also statistically different from that of noise injection.

show abstract

CB3: An Adaptive Error Function for Backpropagation Training

Cited by 12 publications

References 8 publications

Fundamentals of Machine Learning

Fundamentals of Machine Learning

Study of cost functions in Three Term Backpropagation for classification problems

Injecting Chaos in Feedforward Neural Networks

Contact Info

Product

Resources

About