Adaptive and Optimal Online Linear Regression on ℓ1-Balls

Gerchinovitz, Sébastien; Yu, Jia Yuan

doi:10.1007/978-3-642-24412-4_11

Cited by 9 publications

(15 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the EG ± algorithm Kivinen and Warmuth (1997) uses an exponential update rule to formulate an online linear regression algorithm which performs comparably to the best linear predictor under sparsity assumptions. The adaptive EG ± algorithm Gerchinovitz and Yu (2011) further proposes a parameter-free version of EG ± where the learning rate η t is updated in an adaptive fashion, and is a decreasing function of time step t.…”

Section: Related Workmentioning

confidence: 99%

“…We repeat this process until the stopping criteria are met. Note that we use the anytime learning rate schedule of Gerchinovitz and Yu (2011), which is a decreasing function of time t (see Appendix C for more details). A summary of the proposed algorithm, which we refer to as Combinatorial Optimization with Monomial Experts (COMEX), is given in Algorithm 1.…”

Section: The Comex Algorithmmentioning

confidence: 99%

“…In Algorithm 1, we use the anytime learning rate schedule of Gerchinovitz and Yu (2011), which is a decreasing function of time t. The learning rate at time step t is given by:…”

Section: B Proof Of Theorem 32mentioning

confidence: 99%

See 2 more Smart Citations

Combinatorial Black-Box Optimization with Expert Advice

Dadkhahi¹,

Shanmugam²,

Ríos³

et al. 2020

Preprint

View full text Add to dashboard Cite

We consider the problem of black-box function optimization over the boolean hypercube. Despite the vast literature on black-box function optimization over continuous domains, not much attention has been paid to learning models for optimization over combinatorial domains until recently. However, the computational complexity of the recently devised algorithms are prohibitive even for moderate numbers of variables; drawing one sample using the existing algorithms is more expensive than a function evaluation for many black-box functions of interest. To address this problem, we propose a computationally efficient model learning algorithm based on multilinear polynomials and exponential weight updates. In the proposed algorithm, we alternate between simulated annealing with respect to the current polynomial representation and updating the weights using monomial experts' advice. Numerical experiments on various datasets in both unconstrained and sum-constrained boolean optimization indicate the competitive performance of the proposed algorithm, while improving the computational time up to several orders of magnitude compared to state-of-the-art algorithms in the literature.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: The Comex Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Combinatorial Black-Box Optimization with Expert Advice

Dadkhahi¹,

Shanmugam²,

Ríos³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…We assume that the set of outcomes Y is a bounded set, a restriction that can be removed by standard truncation arguments (see e.g. [12]). Let X be some set of covariates, and let F be a class of functions X → Y for some Y ⊆ R. Recall the protocol of the online prediction problem: On each round t ∈ {1, .…”

Section: Assumptions and Definitionsmentioning

confidence: 99%

Online Nonparametric Regression with General Loss Functions

Rakhlin¹,

Sridharan²

2015

Preprint

View full text Add to dashboard Cite

This paper establishes minimax rates for online regression with arbitrary classes of functions and general losses. 1 We show that below a certain threshold for the complexity of the function class, the minimax rates depend on both the curvature of the loss function and the sequential complexities of the class. Above this threshold, the curvature of the loss does not affect the rates. Furthermore, for the case of square loss, our results point to the interesting phenomenon: whenever sequential and i.i.d. empirical entropies match, the rates for statistical and online learning are the same.In addition to the study of minimax regret, we derive a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online prediction algorithms that can be computationally efficient for certain problems. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression.

show abstract

“…Remark 3 In Theorem 2 above, we assumed that the observations y t and the predictions f (x t ) are all bounded by B, and that B is known in advance by the forecaster. We can actually remove this requirement by using adaptive techniques of Gerchinovitz and Yu (2014), namely, adaptive clipping of the intermediate predictions f t,j (x t ) and adaptive Lipschitzification of the square loss functions Remark 4 Even in the case when B is known by the forecaster, the clipping and Lipschitzification techniques of Gerchinovitz and Yu (2014) can be useful to get smaller constants in the regret bound. We could indeed replace the constants 50 and 120 with 8 and 48 respectively.…”

Section: The Chaining Exponentially Weighted Average Forecastermentioning

confidence: 99%

A Chaining Algorithm for Online Nonparametric Regression

Gaillard,

Gerchinovitz

2015

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem of online nonparametric regression with arbitrary deterministic sequences. Using ideas from the chaining technique, we design an algorithm that achieves a Dudley-type regret bound similar to the one obtained in a non-constructive fashion by Rakhlin and Sridharan (2014). Our regret bound is expressed in terms of the metric entropy in the sup norm, which yields optimal guarantees when the metric and sequential entropies are of the same order of magnitude. In particular our algorithm is the first one that achieves optimal rates for online regression over Hölder balls. In addition we show for this example how to adapt our chaining algorithm to get a reasonable computational efficiency with similar regret guarantees (up to a log factor).

show abstract

Adaptive and Optimal Online Linear Regression on ℓ1-Balls

Cited by 9 publications

References 14 publications

Combinatorial Black-Box Optimization with Expert Advice

Combinatorial Black-Box Optimization with Expert Advice

Online Nonparametric Regression with General Loss Functions

A Chaining Algorithm for Online Nonparametric Regression

Contact Info

Product

Resources

About