A new error criterion for posterior probability estimation with neural nets

Abstract

It has been shown recently that neural nets, when trained using the least squares error criterion with a desired output of 1 for belonging to a class and 0 otherwise, produce as their output an estimate of the posterior probability of the class given the input. In this paper, we introduce a new error criterion for training which improves the performance of neural nets as posterior probability estimators, when compared to using least squares. The new criterion is similar to the Kullback-Leibler information measure and is simple to use. We describe a straightforward iterative algorithm for the minimization of the new error criterion, which has been shown to have good convergence properties. Experimental results comparing least squares with the new criterion clearly demonstrate the superiority of the latter for posterior probability estimation.

Cost functions to estimate a posteriori probabilities in multiclass problems

“…The requirements of the application can also influence the cost selection. For instance, when relative differences are more relevant than absolute differences, cross entropy could be preferred to quadratic cost, as the simulations reported in [19] show. A large comparative work has been carried out in the literature showing the advantages of the cross entropy over the quadratic cost.…”

“…A cost function is said to be SSB if has a unique minimum in when the outputs are the a posteriori probabilities of class 1, i.e., (10) According to the above definitions, an SSB detector requires an SSB cost function. For instance, the well-known crossentropy (logarithmic cost) function given by (11) is SSB [19], [20]. Not every cost function is SSB, for instance, an norm is SSB only for [4], [21].…”

“…Theorem 3: A symmetric and separable cost function is SSB iff it can be written in the form (19) where is any positive function , which does not depend on , and is an arbitrary function which does not depend on .…”

“…This is a well known factor responsible for slow training when the quadratic cost function is used , since the adaptation is slow when is near zero or one (for example, see [19] or [25]). This is one of the justifications for using cross-entropy.…”

Radar detection with the Neyman–Pearson criterion using supervised-learning-machines trained with the cross-entropy error

“…The algorithm used for training with the Cross-Entropy error is the one described in [43], while the algorithm used for training with the Sum-of-Squares error and the Minkowski error (R = 1), is the Conjugate Gradient method [44].…”