Beyond the Oracle: Opportunities of Piecewise Differentiation

Griewank, Andreas; Walther, Andrea

doi:10.1007/978-3-030-34910-3_10

Cited by 3 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, exactly this is simply assumed in the frequently invoked oracle paradigm, which states that at any point x P R n the function value f pxq and an element g P B f pxq can be evaluated. We have argued in [18] that this is not at all a reasonable assumption.…”

Section: Computation Of Generalized Gradients and Constructive Oracle Paradigmmentioning

confidence: 97%

“…i.e., Equations ( 16) and (18). As can be seen, the matricesM andL have the required strictly lower triangular form.…”

Section: Forming and Extending The Abs-linear Formmentioning

confidence: 99%

“…Global optimality can be characterized theoretically in terms of ε´subgradients, albeit with ε arbitrarily large [26]. There is the possibility that the alternative definition of ε-gradients given in [18] might allow one to constructively check for global optimality. It does not really seem clear how these global optimality conditions can be used to derive corresponding algorithms.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Polyhedral DC Decomposition and DCA Optimization of Piecewise Linear Functions

Griewank

Walther

2020

Algorithms

Self Cite

View full text Add to dashboard Cite

For piecewise linear functions f : R n ↦ R we show how their abs-linear representation can be extended to yield simultaneously their decomposition into a convex f and a concave part f ^ , including a pair of generalized gradients g ∈ R n ∋ g ^ . The latter satisfy strict chain rules and can be computed in the reverse mode of algorithmic differentiation, at a small multiple of the cost of evaluating f itself. It is shown how f and f ^ can be expressed as a single maximum and a single minimum of affine functions, respectively. The two subgradients g and − g ^ are then used to drive DCA algorithms, where the (convex) inner problem can be solved in finitely many steps, e.g., by a Simplex variant or the true steepest descent method. Using a reflection technique to update the gradients of the concave part, one can ensure finite convergence to a local minimizer of f, provided the Linear Independence Kink Qualification holds. For piecewise smooth objectives the approach can be used as an inner method for successive piecewise linearization.

show abstract

Section: Computation Of Generalized Gradients and Constructive Oracle Paradigmmentioning

confidence: 97%

“…i.e., Equations ( 16) and (18). As can be seen, the matricesM andL have the required strictly lower triangular form.…”

Section: Forming and Extending The Abs-linear Formmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Polyhedral DC Decomposition and DCA Optimization of Piecewise Linear Functions

Griewank

Walther

2020

Algorithms

Self Cite

View full text Add to dashboard Cite

show abstract

“…The quest for computationally cheap nonsmooth derivative has a long history in AD litterature. Existing works of Griewank [33,30,28,29] are essentially based on piecewise smoothness structures [52]. A cheap subgradient principle was also given in [34], but it requires a very strong qualification condition.…”

Section: Contributionsmentioning

confidence: 99%

On the complexity of nonsmooth automatic differentiation

Bolte¹,

Ryan²,

Pauwels³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs. Prominent examples are the famous relu and convolutional neural networks together with their standard loss functions. Using the recent notion of conservative gradients, we then establish a "nonsmooth cheap gradient principle" for backpropagation encompassing most concrete applications. Nonsmooth backpropagation's cheapness contrasts with concurrent forward approaches which have, at this day, dimensional-dependent worst case estimates. In order to understand this class of methods, we relate the complexity of computing a large number of directional derivatives to that of matrix multiplication. This shows a fundamental limitation for improving forward AD for that task. Finally, while the fastest algorithms for computing a Clarke subgradient are linear in the dimension, it appears that computing two distinct Clarke (resp. lexicographic) subgradients for simple neural networks is NP-Hard.

show abstract