Computational concepts in classification: Neural networks, statistical pattern recognition, and model-based vision

Abstract

Abstract. A large number of algorithms have been developed for classification and recognition. These algorithms can be divided into three major paradigms: statistical pattern recognition, neural networks, and model-based vision. Neural networks embody an especially rich field of approaches based on a variety of architectures, learning mechanisms, biological and algorithmic motivations, and application areas. Mathematical analysis of these approaches and paradigms reveals that there are only a few computational concepts permeating all the diverse approaches and serving as a basis for all paradigms and algorithms for classification and recognition.These basic computational concepts are reviewed in this paper with the purposes of (i) providing a mathematical continuity to seemingly disparate techniques, (ii) establishing basic mathematical limitations on applicability of existing techniques, (iii) discerning fundamental questions facing the classification field, and (iv) searching for directions in which answers to these questions may be found.

Neurally and Mathematically Motivated Architecture for Language and Thought

“…Mathematically, CC is related to formal logic, which turned out to be used by most mathematical procedures, even by those specifically designed to overcome logic limitations, such as neural networks and fuzzy logic [25,29-31,64]. The mathematics capable of overcoming CC, dynamic logic (DL), which models the process of satisfaction of KI, while overcoming CC has been developed in [13,23-25,32-34]. In several cases it was mathematically proved that DL achieves the best possible performance [35-38].…”

Conundrum of combinatorial complexity

“…Nonparametric paradigms (discriminating surfaces and nearest neighbors) have been used to surpass the limitations of simple parametric methods. However, due to the fact that the volume of a classification space grows exponentially with the dimensionality (number of features), training requirements for nonparametric paradigms are often exponential in terms of the problem complexity [36]. This is essentially the same problem that was encountered earlier in the field of adaptive control and was named "the curse of dimensionality" [4].…”

“…This relationship has been discussed recently for geometric patterns and for function approximation [14], [36]. The issue of the roles of a priori knowledge vs. adaptive learning has been of an overriding concern in the research of mathematics of intelligence since its inception.…”

“…There are three basic approaches used to partition a classification space among classes: discriminating surfaces, nearest neighbors, and model-based parametric estimation [36]. Early parametric methods based on relatively simple probabil- istic models such as Gaussian distributions were limited to simple classifier shapes (such as the quadratic classifier).…”

“…A mathematical analysis of existing approaches to the design of systems and algorithms of mathematical intelligence leads to the conclusion that computational concepts of most of today's neural networks originate in pattern recognition algorithms, and that there are four basic concepts forming the foundation for all the multiplicity of existing algorithms and neural networks [35], [36], [37]. These are 1) the concept of rule-based systems [28], defined by the factor of a priority; 2) the concept of nearest neighbors and 3) the concept of discriminating surfaces, both defined by the factor of adaptivity [12], [33]; and 4) the concept of parametric models [41], [48], that attempts to combine a priority and adaptivity.…”