Branch-and-bound performance estimation programming: a unified methodology for constructing optimal optimization methods

Gupta, Shuvomoy Das; Parys, Bart Van; Ryu, Ernest K.

doi:10.1007/s10107-023-01973-1

Cited by 9 publications

(2 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is due to our recursive construction, and is directly evident from the aforementioned fact that the i-th stepsize depends on the sparsity pattern of the binary expansion of i. This fractal structure aligns with the numerical observations in [19,32], and is in stark contrast with all classical stepsize schedules which, if timevarying, decay monotonically in the iteration number i, e.g., as 1/i. Approximate periodicity.…”

Section: Discussion Of Silver Stepsize Schedulesupporting

confidence: 76%

“…A key difficulty in extending this to larger horizons n is that the search for optimal stepsizes is non-convex. In 2022, Das Gupta et al [19] combined Branch & Bound techniques with the PESTO SDP to develop algorithms that perform this search numerically, and as an example used this to compute good approximate schedules in the convex setting for larger values of n up to 50. Grimmer [32] very recently developed a technique to round these Branch & Bound solutions to exact rational certificates.…”

Section: The General Case Of Convex Optimizationmentioning

confidence: 99%

See 1 more Smart Citation

Kernel Approximation on Algebraic Varieties

Altschuler¹,

Parrilo²

2023

SIAM J. Appl. Algebra Geometry

View full text Add to dashboard Cite

Can we accelerate convergence of gradient descent without changing the algorithm-just by carefully choosing stepsizes? Surprisingly, we show that the answer is yes. Our proposed Silver Stepsize Schedule optimizes strongly convex functions in κ log ρ 2 ≈ κ 0.7864 iterations, where ρ = 1 + √ 2 is the silver ratio and κ is the condition number. This is intermediate between the textbook unaccelerated rate κ and the accelerated rate κ 1/2 due to Nesterov in 1983. The non-strongly convex setting is conceptually identical, and standard black-box reductions imply an analogous partially accelerated rate ε − log ρ 2 ≈ ε −0.7864 . We conjecture and provide partial evidence that these rates are optimal among all stepsize schedules.The Silver Stepsize Schedule is constructed recursively in a fully explicit way. It is nonmonotonic, fractal-like, and approximately periodic of period κ log ρ 2 . This leads to a phase transition in the convergence rate: initially super-exponential (acceleration regime), then exponential (saturation regime).The core algorithmic intuition is hedging between individually suboptimal strategies-short steps and long steps-since bad cases for the former are good cases for the latter, and vice versa. Properly combining these stepsizes yields faster convergence due to the misalignment of worst-case functions. The key challenge in proving this speedup is enforcing long-range consistency conditions along the algorithm's trajectory. We do this by developing a technique that recursively glues constraints from different portions of the trajectory, thus removing a key stumbling block in previous analyses of optimization algorithms. More broadly, we believe that the concepts of hedging and multi-step descent have the potential to be powerful algorithmic paradigms in a variety of contexts in optimization and beyond.This series of papers publishes and extends the first author's 2018 Master's Thesis (advised by the second author)-which established for the first time that carefully choosing stepsizes can enable acceleration in convex optimization. Prior to this thesis, the only such result was for the special case of quadratic optimization, due to Young in 1953.

show abstract

Section: Discussion Of Silver Stepsize Schedulesupporting

confidence: 76%

Section: The General Case Of Convex Optimizationmentioning

confidence: 99%

Kernel Approximation on Algebraic Varieties

Altschuler¹,

Parrilo²

2023

SIAM J. Appl. Algebra Geometry

View full text Add to dashboard Cite

show abstract

Branch Definition–Based Modeling for Sustainable Supply Chain Management in Regional Digital Economy

Qiu

2024

Process Integr Optim Sustain

View full text Add to dashboard Cite

Evolving scientific discovery by unifying data and background knowledge with AI Hilbert

Cory-Wright,

Cornelio,

Dash

et al. 2024

Nat Commun

View full text Add to dashboard Cite

The discovery of scientific formulae that parsimoniously explain natural phenomena and align with existing background theory is a key goal in science. Historically, scientists have derived natural laws by manipulating equations based on existing knowledge, forming new equations, and verifying them experimentally. However, this does not include experimental data within the discovery process, which may be inefficient. We propose a solution to this problem when all axioms and scientific laws are expressible as polynomials and argue our approach is widely applicable. We model notions of minimal complexity using binary variables and logical constraints, solve polynomial optimization problems via mixed-integer linear or semidefinite optimization, and prove the validity of our scientific discoveries in a principled manner using Positivstellensatz certificates. We demonstrate that some famous scientific laws, including Kepler’s Law of Planetary Motion and the Radiated Gravitational Wave Power equation, can be derived in a principled manner from axioms and experimental data.

show abstract

Branch-and-bound performance estimation programming: a unified methodology for constructing optimal optimization methods

Cited by 9 publications

References 55 publications

Kernel Approximation on Algebraic Varieties

Kernel Approximation on Algebraic Varieties

Branch Definition–Based Modeling for Sustainable Supply Chain Management in Regional Digital Economy

Evolving scientific discovery by unifying data and background knowledge with AI Hilbert

Contact Info

Product

Resources

About