Decision boundaries and convex hulls in the feature space that deep learning functions learn from images

Abstract

The success of deep neural networks in image classification and learning can be partly attributed to the features they extract from images. It is often speculated about the properties of a low-dimensional manifold that models extract and learn from images. However, there is not sufficient understanding about this low-dimensional space based on theory or empirical evidence. For image classification models, their last hidden layer is the one where images of each class is separated from other classes and it also has the least number of features. Here, we develop methods and formulations to study that feature space for any model. We study the partitioning of the domain in feature space, identify regions guaranteed to have certain classifications, and investigate its implications for the pixel space. We observe that geometric arrangements of decision boundaries in feature space is significantly different compared to pixel space, providing insights about adversarial vulnerabilities, image morphing, extrapolation, ambiguity in classification, and the mathematical understanding of image classification models.

Over-parameterization: A Necessary Condition for Models that Extrapolate

“…Moreover, adversarial inputs are recognizably close to decision boundaries in the feature space. This way, adversarial inputs can be detected based on their closeness to decision boundaries of feature space [13]. However, in the pixel space, closeness to decision boundaries is not useful for detecting adversarial inputs.…”