Foundations of Support Constraint Machines

Abstract

The mathematical foundations of a new theory for the design of intelligent agents are presented. The proposed learning paradigm is centered around the concept of constraint, representing the interactions with the environment, and the parsimony principle. The classical regularization framework of kernel machines is naturally extended to the case in which the agents interact with a richer environment, where abstract granules of knowledge, compactly described by different linguistic formalisms, can be translated into the unified notion of constraint for defining the hypothesis set. Constrained variational calculus is exploited to derive general representation theorems that provide a description of the optimal body of the agent (i.e., the functional structure of the optimal solution to the learning problem), which is the basis for devising new learning algorithms. We show that regardless of the kind of constraints, the optimal body of the agent is a support constraint machine (SCM) based on representer theorems that extend classical results for kernel machines and provide new representations. In a sense, the expressiveness of constraints yields a semantic-based regularization theory, which strongly restricts the hypothesis set of classical regularization. Some guidelines to unify continuous and discrete computational mechanisms are given so as to accommodate in the same framework various kinds of stimuli, for example, supervised examples and logic predicates. The proposed view of learning from constraints incorporates classical learning from examples and extends naturally to the case in which the examples are subsets of the input space, which is related to learning propositional logic clauses.

Learning in Variable-Dimensional Spaces

“…1, within the framework of regularization. We distinguish functions on LB-patterns (1,4,6,7,8) with respect to functions on FB-patterns (2,3,5,9). In the first case 2, 3, 4, 5, 6, 7, 8, 9} is the collection of pattern identifiers, while in the second case…”

“…We conjecture that one can approach the proposed formulation of learning by functions in the RKHS of a given kernel, even though the presence of LB-patterns violate classic kernel assumptions. In the rest of this paper, we propose a unified solution based on the general framework of learning from constraint given in [1], which allows us to treat functions on LB-patterns just like those on FB-patterns by distributional differential equations.…”

“…We propose a generalization to multitask learning of what was proposed in [12] for regularization networks with soft-constraints associated with supervised examples, whose definition is rooted in Tikhonov's regularization theory [13] and related studies on spline functions [14], [15]. In the case n = 1, the operator P has been related to the notion of kernel [1], [16]- [18], while in [19], there are some relevant links between regularization networks and kernel machines.…”

“…We present a special form of a general learning framework that is founded on the notion of constraint and of regularization operator [1], [2]. We prove that, under assumptions that are often met in practice, the optimal solution of the stated problem can be given in terms of classic kernelbased representations.…”

“…In Section V, we state the main results concerning the representation of the optimal solution along with algorithmic issues. In Section VI, we give some experimental results 1 and, finally, in Section VII, some conclusions are drawn. Three appendixes are included to support the given theoretical results.…”