Towards Lower Bounds on the Depth of ReLU Neural Networks

Hertrich, Christoph; Basu, Amitabh; Summa, Marco Di; Skutella, Martin

doi:10.1137/22m1489332

Cited by 2 publications

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, many surprisingly basic questions remain open. For example, it is not known whether two layers of ReLU units (with any width) are sufficient to compute the function f : R 4 → R, x → max{0, x 1 , x 2 , x 3 , x 4 } [24,28].…”

Section: Introductionmentioning

confidence: 99%

ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation

Hertrich

Sering

2023

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper studies the expressive power of artificial neural networks with rectified linear units. In order to study them as a model of real-valued computation, we introduce the concept of Max-Affine Arithmetic Programs and show equivalence between them and neural networks concerning natural complexity measures. We then use this result to show that two fundamental combinatorial optimization problems can be solved with polynomial-size neural networks. First, we show that for any undirected graph with n nodes, there is a neural network (with fixed weights and biases) of size O(n 3 ) that takes the edge weights as input and computes the value of a minimum spanning tree of the graph. Second, we show that for any directed graph with n nodes and m arcs, there is a neural network of size O(m 2 n 2 ) that takes the arc capacities as input and computes a maximum flow. Our results imply that these two problems can be solved with strongly polynomial time algorithms that solely uses affine transformations and maxima computations, but no comparison-based branchings.

show abstract

Section: Introductionmentioning

confidence: 99%