A Universal Approximation Result for Difference of Log-Sum-Exp Neural Networks

Abstract

We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node (LSE networks) is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of networks maps to a family of subtractionfree ratios of generalized posynomials, which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main advantage of Difference-LSE networks with respect to classical feedforward neural networks is that, after a standard training phase, they provide surrogate models for design that possess a specific differenceof-convex-functions form, which makes them optimizable via relatively efficient numerical methods. In particular, by adapting an existing difference-of-convex algorithm to these models, we obtain an algorithm for performing effective optimization-based design. We illustrate the proposed approach by applying it to data-driven design of a diet for a patient with type-2 diabetes.

Tropical Geometry and Machine Learning

“…In any adjunction, δ is a dilation and ε is an erosion. The double inequality ( 19) is equivalent to the inequality (17) satisfied by a residuation pair of increasing operators if we identify the residuated map ψ with δ and its residual ψ with ε. Furthermore, from (19) or (17), it follows that any adjunction (δ, ε) automatically yields an opening α = δε and a closing β = εδ, where the composition of two operators is written as an operator product.…”

“…The double inequality ( 19) is equivalent to the inequality (17) satisfied by a residuation pair of increasing operators if we identify the residuated map ψ with δ and its residual ψ with ε. Furthermore, from (19) or (17), it follows that any adjunction (δ, ε) automatically yields an opening α = δε and a closing β = εδ, where the composition of two operators is written as an operator product. Viewing (δ, ε) as an adjunction instead of a residuation pair has the advantage of the additional geometrical intuition and visualization afforded by the dilation and erosion operators in image and shape analysis.…”

“…Early connections between tropical geometry and neural networks were sketched in [18] and later developed in greater detail in [19] and [114]. Tools from tropical geometry (in particular, the Maslov dequantization) have also been used to design neural networks that approximate convex and log-log convex data [16] and general continuous functions over convex sets [17]. For the remainder of the section, we primarily develop the tropical-geometric characterization of neural network layers following [19] and [114] and describe other applications near its end.…”

“…The rank K of the model can be increased until some error threshold is reached. Interesting and promising generalizations of the above max-affine representation for convex functions include works that use softmax instead of max, via the log-sum-exp models for convex and log-log convex data [16], [17], [53]. Other iterative approaches for convex PWL data fitting include [108].…”

Advances in Morphological Neural Networks: Training, Pruning and Enforcing Shape Constraints

“…As h → 0, the soft versions approach the hard ones: lim h→0 x ∨ h y = x ∨ y and lim h→0 x ∧ h y = x ∧ y. For small positive values of h, these approximations are part of the Log-Sum-Exp family, used in convex analysis and recently linked to tropical polynomials [25,26]. We use the reciprocal of h, the hardness parameter β = h −1 .…”