Adaptive sequential learning

Wilson, Craig; Veeravalli, Venugopal V.

doi:10.1109/acssc.2016.7869052

Cited by 1 publication

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have demonstrated that assumption C.1 holds for the direct estimate of ρ under ( 12) and (2). Note that whether we assume (12) or (2) does not matter for analysis.…”

Section: E Change In Minimizers Unknownmentioning

confidence: 77%

“…This is a smoothed version of the hinge loss used in support vector machines (SVM) [33]. We suppose that at time n, the two classes have features drawn from a Gaussian distribution with covariance matrix σ 2 I but different means µ (1) n and µ (2) n , i.e.,…”

Section: B Synthetic Classificationmentioning

confidence: 99%

“…Under this model, we find approximate minimizers w n of each function f n (w) by drawing K n samples {z n (k)} K n k=1 iid ∼ p n at time n. We do not make any assumptions about the Research reported in the paper was supported by the NSF under award CCF 11-11342, and by the Army Research Laboratory under Cooperative Agreement W911NF-17-2-0196, through the University of Illinois at Urbana-Champaign. Part of this work was presented in ICASSP 2016 [1] and Asilomar Conference 2016 [2]. Craig Wilson is now at Google.…”

Section: Introductionmentioning

confidence: 99%

“…We also consider extensions relevant to the machine learning setting, including a cost-based approach to select the number of samples with a cost budget over a fixed horizon, and an approach to applying cross-validation for model selection. Some of the results in this paper have reported in conference publications [1] and [2], which do not contain proofs of the key results due to space limitations. Moreover, we provide substantially more detailed numerical results and simulations in this paper than those given in [1] and [2].…”

Section: Introductionmentioning

confidence: 99%

“…Some of the results in this paper have reported in conference publications [1] and [2], which do not contain proofs of the key results due to space limitations. Moreover, we provide substantially more detailed numerical results and simulations in this paper than those given in [1] and [2].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Adaptive sequential machine learning

Wilson

Veeravalli

2019

Sequential Analysis

Self Cite

View full text Add to dashboard Cite

A framework previously introduced in [3] for solving a sequence of stochastic optimization problems with bounded changes in the minimizers is extended and applied to machine learning problems such as regression and classification. The stochastic optimization problems arising in these machine learning problems is solved using algorithms such as stochastic gradient descent (SGD). A method based on estimates of the change in the minimizers and properties of the optimization algorithm is introduced for adaptively selecting the number of samples at each time step to ensure that the excess risk, i.e., the expected gap between the loss achieved by the approximate minimizer produced by the optimization algorithm and the exact minimizer, does not exceed a target level. A bound is developed to show that the estimate of the change in the minimizers is nontrivial provided that the excess risk is small enough. Extensions relevant to the machine learning setting are considered, including a cost-based approach to select the number of samples with a cost budget over a fixed horizon, and an approach to applying cross-validation for model selection. Finally, experiments with synthetic and real data are used to validate the algorithms.

show abstract

“…We have demonstrated that assumption C.1 holds for the direct estimate of ρ under ( 12) and (2). Note that whether we assume (12) or (2) does not matter for analysis.…”

Section: E Change In Minimizers Unknownmentioning

confidence: 77%

Section: B Synthetic Classificationmentioning

confidence: 99%