On the Weight of Halfspaces over Hamming Balls

Abstract

For S ⊆ {0, 1}n , a Boolean function f : S → {−1, 1} is a halfspace over S if there exist w ∈ R n and θ ∈ R such that f (x) = sign(w · x − θ) for all x ∈ S. We give bounds on the size of integer weights w 1 , . . . , w n ∈ Z that are required to represent halfspaces over Hamming balls S = {x ∈ {0, 1} n : x 1 + · · · + x n ≤ k}. Such weight bounds for halfspaces over Hamming balls have immediate consequences for the performance of learning algorithms in the common scenario of learning from very high-dimensional categorical examples which are such that only a small number of features are active in each example.We give upper and lower bounds on weight both for exact representation (when sign(w · x−θ) must equal f (x) for every x ∈ S) and for ε-approximate representation (when sign(w · x−θ) may disagree with f (x) for up to an ε fraction of points x ∈ S). Our results show that extremal bounds for exact representation are qualitatively rather similar whether the domain is all of {0, 1} n or the Hamming ball {0, 1} n ≤k , but extremal bounds for approximate representation are qualitatively very different between these two domains.

Polynomial Threshold Functions for Decision Lists

“…The case of Hamming balls {0, 1} n ≤k consisting of all vectors with at most k 1s received some attention. Long and Servedio [21] gave bounds for the weights of PTFs for degree d = 1. Their main motivation to study this setting comes from learning theory: in scenarios involving learning a categorical data the common representation for examples in the one-hot encoded vector, which might have an extremely large amount of features, but only a small faction of them can be active at the same time.…”