The Geometry of Deep Networks: Power Diagram Subdivision

Abstract

We study the geometry of deep (neural) networks (DNs) with piecewise affine and convex nonlinearities. The layers of such DNs have been shown to be max-affine spline operators (MASOs) that partition their input space and apply a regiondependent affine mapping to their input to produce their output. We demonstrate that each MASO layer's input space partitioning corresponds to a power diagram (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons). We further show that a composition of MASO layers (e.g., the entire DN) produces a progressively subdivided power diagram and provide its analytical form. The subdivision process constrains the affine maps on the (exponentially many) power diagram regions to greatly reduce their complexity. For classification problems, we obtain a formula for a MASO DN's decision boundary in the input space plus a measure of its curvature that depends on the DN's nonlinearities, weights, and architecture. Numerous numerical experiments support and extend our theoretical results.Preprint. Under review.

Max-Affine Spline Insights Into Deep Network Pruning

“…Whenever a DN can be expressed as a MASO, it becomes a continuous piecewise affine mapping (Unser, 2018) with an inherent input space partition and perregion affine mappings. In fact, by the form of (1) it is clear that there exists an underlying layer input space partition based on the realization of the maximum operator; for a study of this partitioning, see (Montufar et al, 2014;Balestriero et al, 2019).…”

“…As we also studied in the previous section, for classification performances, not all the input space partition regions and boundaries are relevant since not all affect the final decision boundary. Knowing a priori which regions of the input space partition are helping in solving the task is extremely challenging since it requires knowledge of the decision boundary and of the input space partition, both being highly difficult to obtain for high dimensional spaces and large networks (Montufar et al, 2014;Balestriero et al, 2019;Hanin & Rolnick, 2019). We propose in this section an alternative method leveraging the spline formulation and providing a pruning strategy based on measuring how different units can be seen as redundant in term of their contribution to the entire DN input space partition.…”

“…The above result is crucial as removing units that do not affect the layer input space is synonym of removing units that do not affect the entire DN input space partition, see (Balestriero et al, 2019) for details. In practice, units with small enough but nonzero N ρ (k, k ′ ) are also highly redundant and can be removed.…”