Active Learning and Basis Selection for Kernel-Based Linear Models: A Bayesian Perspective

Abstract

We develop an active learning algorithm for kernel-based linear regression and classification. The proposed greedy algorithm employs a minimum-entropy criterion derived using a Bayesian interpretation of ridge regression. We assume access to a matrix, Φ ∈ R N ×N , for which the (i, j) th element is defined by the kernel function K(γ i , γ j ) ∈ R, with the observed datawhere y i is a real-valued response or integer-valued label, which we do not have access to a priori. To achieve this goal, a sub-matrix, Φ I l ,I b ∈ R n×m , is sought that corresponds to the intersection of n rows and m columns of Φ, indexed by the sets I l and I b respectively. Typically m N and n N . and (ii) using active learning, sequentially determine which subset of n elements of {y i } N i=1 should be acquired; both stopping values, |I b | = m and |I l | = n, are also to be inferred from the data. These steps are taken with the goal of minimizing the uncertainty of the model parameters, x, as measured by the differential entropy of its posterior distribution. The parameter vector x ∈ R m , as well as the model bias η ∈ R, is then learned from the resulting problem, y I l = Φ I l ,I b x + η1 + . The remaining N − n responses/labels not included in y I l can be inferred by applying x to the remaining N − n rows of Φ :,I b .We show experimental results for several regression and classification problems, and compare with other active learning methods.

A Bayesian Active Learning Framework for a Two-Class Classification Problem

“…The synthetic data set, due to Paisley [6], consists of 200 observations, 100 from each one of the two classes, in a bi-dimensional space. The data, plotted in figure 1, is composed of two classes defined by two manifolds, which are not linearly separable in this bi-dimensional space.…”

“…To select this new feature vector, in this paper, we propose to maximize the difference between the entropies of the posterior distribution before and after adding the new feature vector (see [6,9]) to obtain…”

“…In [6], a Bayesian active learning procedure for finite dimensional feature spaces is proposed. Assuming that φ(x i ), i = 1, .…”

“…, P is given by φ(x i ) . Then, a subset of size C of the L columns of Φ :,: , denoted by Φ :,IC , is selected using the differential entropy instead of the response functions y(x i ) [6]. Notice that this approach is in contrast to other basis selection techniques which make explicit use of the response functions, for example, [3] in the context of SVM, [4] in the context of sparse representation, and [1] considering compressive sensing.…”

“…Notice that this approach is in contrast to other basis selection techniques which make explicit use of the response functions, for example, [3] in the context of SVM, [4] in the context of sparse representation, and [1] considering compressive sensing. To select the rows of Φ :,IC , for which the response associated to φ(x i ) will be queried, a criterion based again on differential entropy is utilized (see [6] for details). See also [5] for the general theory and [9] for the use of the approach in compressive sensing.…”