Regret Analysis of Global Optimization in Univariate Functions with Lipschitz Derivatives

Gokcesu, Kaan; Gokcesu, Hakan

doi:10.48550/arxiv.2108.10859

Cited by 10 publications

(28 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Brent [47] proposes a variant where the function f (•) is required to be defined on a compact interval with a bounded second derivative. The work in [48] studies the problem with a more generalized Lipschitz regularity. For a more detailed perspective, the book of Horst and Tuy [49] has a general discussion about the role of deterministic algorithms in global optimization.…”

Section: B Related Workmentioning

confidence: 99%

“…The work of Bouttier et al [52] studies the regret bounds of Piyavskii-Shubert algorithm under noisy evaluations. Instead of the weaker simple regret, the work in [48] provides cumulative regret bounds for variants of Piyavskii-Shubert algorithm. Although [48] shows that Piyavskii-Shubert algorithm has nice regret bounds for a variety of different regularity conditions (e.g., Lipschitz continuity and smoothness), the optimization of the lower bounding proxy functions themselves are not always easy to solve.…”

Section: B Related Workmentioning

confidence: 99%

“…Instead of the weaker simple regret, the work in [48] provides cumulative regret bounds for variants of Piyavskii-Shubert algorithm. Although [48] shows that Piyavskii-Shubert algorithm has nice regret bounds for a variety of different regularity conditions (e.g., Lipschitz continuity and smoothness), the optimization of the lower bounding proxy functions themselves are not always easy to solve. When the operational cost is not only driven by the function evaluation but also selecting queries, alternative approaches are need.…”

Section: B Related Workmentioning

confidence: 99%

“…To this end, we define a regularity measure. Instead of the restrictive Lipschitz continuity or smoothness [48]; we define a weaker, more general regularity condition.…”

Section: Contributions and Organizationmentioning

confidence: 99%

“…where Γ(•) is some function. One such algorithm is the famous Piyavskii-Shubert algorithm (and its variants) [48].…”

Section: Contributions and Organizationmentioning

confidence: 99%

See 4 more Smart Citations

Low Regret Binary Sampling Method for Efficient Global Optimization of Univariate Functions

Gokcesu¹,

Gokcesu²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we propose a computationally efficient algorithm for the problem of global optimization in univariate loss functions. For the performance evaluation, we study the cumulative regret of the algorithm instead of the simple regret between our best query and the optimal value of the objective function. Although our approach has similar regret results with the traditional lower-bounding algorithms such as the Piyavskii-Shubert method for the Lipschitz continuous or Lipschitz smooth functions, it has a major computational cost advantage. In Piyavskii-Shubert method, for certain types of functions, the query points may be hard to determine (as they are solutions to additional optimization problems). However, this issue is circumvented in our binary sampling approach, where the sampling set is predetermined irrespective of the function characteristics. For a search space of [0, 1], our approach has at most L log(3T ) and 2.25H regret for L-Lipschitz continuous and H-Lipschitz smooth functions respectively. We also analytically extend our results for a broader class of functions that covers more complex regularity conditions.

show abstract

Section: B Related Workmentioning

confidence: 99%

Section: B Related Workmentioning

confidence: 99%

Section: B Related Workmentioning

confidence: 99%

“…To this end, we define a regularity measure. Instead of the restrictive Lipschitz continuity or smoothness [48]; we define a weaker, more general regularity condition.…”

Section: Contributions and Organizationmentioning

confidence: 99%

“…where Γ(•) is some function. One such algorithm is the famous Piyavskii-Shubert algorithm (and its variants) [48].…”

Section: Contributions and Organizationmentioning

confidence: 99%

See 3 more Smart Citations

Low Regret Binary Sampling Method for Efficient Global Optimization of Univariate Functions

Gokcesu¹,

Gokcesu²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Sequential Outlier Detection Based on Incremental Decision Trees

Gokcesu

Neyshabouri

Gokcesu

et al. 2019

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

We introduce an online outlier detection algorithm to detect outliers in a sequentially observed data stream. For this purpose, we use a two-stage filtering and hedging approach. In the first stage, we construct a multimodal probability density function to model the normal samples. In the second stage, given a new observation, we label it as an anomaly if the value of aforementioned density function is below a specified threshold at the newly observed point. In order to construct our multimodal density function, we use an incremental decision tree to construct a set of subspaces of the observation space. We train a single component density function of the exponential family using the observations, which fall inside each subspace represented on the tree. These single component density functions are then adaptively combined to produce our multimodal density function, which is shown to achieve the performance of the best convex combination of the density functions defined on the subspaces. As we observe more samples, our tree grows and produces more subspaces. As a result, our modeling power increases in time, while mitigating overfitting issues. In order to choose our threshold level to label the observations, we use an adaptive thresholding scheme. We show that our adaptive threshold level achieves the performance of the optimal prefixed threshold level, which knows the observation labels in hindsight. Our algorithm provides significant performance improvements over the state of the art in our wide set of experiments involving both synthetic as well as real data.

show abstract

Survey of Optimization Algorithms in Modern Neural Networks

Abdulkadirov¹,

Lyakhov²,

Nagornov³

2023

Preprint

View full text Add to dashboard Cite

Creating self-learning algorithms, developing deep neural networks and improving other methods that "learn" for various areas of human activity is the main goal of the theory of machine learning. It helps to replace the human with a machine, aiming to increase the quality of production. The theory of artificial neural networks, which already have replaced the humans in problems of detection of moving objects, recognition of images or sounds, time series prediction, big data analysis and numerical methods remains the most dispersed branch of the theory of machine learning. Certainly, for each area of human activity it is necessary to select appropriate neural network architectures, methods of data processing and some novel tools from applied mathematics. But the universal problem for all these neural networks with specific data is the achieving the highest accuracy in short time. Such problem can be resolved by increasing sizes of architectures and improving data preprocessing, where the accuracy rises with the training time. But there is a possibility to increase the accuracy without time growing, applying optimization methods. In this survey we demonstrate existing optimization algorithms of all types, which can be used in neural networks. There are presented modifications of basic optimization algorithms, such as stochastic gradient descent, adaptive moment estimation, Newton and quasi-Newton optimization methods. But the most recent optimization algorithms are related to information geometry, for Fisher-Rao and Bregman metrics. This approach in optimization extended the theory of classic neural networks to quantum and complex-valued neural networks, due to geometric and probabilistic tools. There are provided applications of all introduced optimization algorithms, what delighted many kinds of neural networks, which can be improved by including any advanced approaches in minimization of the loss function. Afterwards, we demonstrated ways of developing optimization algorithms in further researches, engaging neural networks with progressive architectures. Classical gradient based optimizers can be replaced by fractional order, bilevel and, even, gradient free optimization methods. There is a possibility to add such analogues in graph, spiking, complex-valued, quantum and wavelet neural networks. Besides the usual problems of image recognition, time series prediction, object detection, there are many are other tasks for modern theory of machine learning, such as solving problem of quantum computations, partial differential and integro-differential equations, stochastic processes and Brownian motion, making decisions and computer algebra.

show abstract

Regret Analysis of Global Optimization in Univariate Functions with Lipschitz Derivatives

Cited by 10 publications

References 31 publications

Low Regret Binary Sampling Method for Efficient Global Optimization of Univariate Functions

Low Regret Binary Sampling Method for Efficient Global Optimization of Univariate Functions

Sequential Outlier Detection Based on Incremental Decision Trees

Survey of Optimization Algorithms in Modern Neural Networks

Contact Info

Product

Resources

About