Sample Complexity for Computational Classification Problems

Abstract

In a statistical setting of the classification (pattern recognition) problem the number of examples required to approximate an unknown labelling function is linear in the VC dimension of the target learning class. In this work we consider the question of whether such bounds exist if we restrict our attention to computable classification methods, assuming that the unknown labelling function is also computable. We find that in this case the number of examples required for a computable method to approximate the labelling function not only is not linear, but grows faster (in the VC dimension of the class) than any computable function. No time or space constraints are put on the predictors or target functions; the only resource we consider is the training examples.The task of classification is considered in conjunction with another learning problem-data compression. An impossibility result for the task of data compression allows us to estimate the sample complexity for pattern recognition.

On-Device Machine Learning: An Algorithms and Learning Theory Perspective

“…is disjunctive in such theories. In fact, the works in [28,104] raise some critical questions about the adequacy of the PAC learning framework for designing ML algorithms with limited computation complexity. These questions have led to a substantial amount of work modifying the PAC learning paradigm to provide an explicit tradeoff between sample and computation complexity.…”

Some Sufficient Conditions on an Arbitrary Class of Stochastic Processes for the Existence of a Predictor

“…The normalized maximum likelihood distribution considered above does not in general lead to the optimum solution for this problem. The optimum solution is obtained through the result that relates the minimax (12) to the so-called channel capacity. For a set A of measures on a finite set X the channel capacity of A is defined as…”

“…Conceptually this is a much simpler problem; however, in some cases it can be intractable (see e.g. [12]). In general, for each particular class of classifiers a separate algorithm should be constructed to find efficiently a classifier that fits the data.…”