Bilevel Optimization and Machine Learning

Bennett, Kristin P.; Kunapuli, Gautam; Hu, Jing; Pang, Jong-Shi

doi:10.1007/978-3-540-68860-0_2

Cited by 45 publications

(42 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We review some methods for the numerical computation ofĝ in (61) relying on the relationship between machine learning and convex optimization (Bennett & Parrado-Hernandez, 2006;Bottou, Chapelle, DeCoste & Weston, 2007;Rockafellar, 1970).…”

Section: Computational Issuesmentioning

confidence: 99%

Kernel methods in system identification, machine learning and function estimation: A survey

et al. 2014

View full text Add to dashboard Cite

a b s t r a c tMost of the currently used techniques for linear system identification are based on classical estimation paradigms coming from mathematical statistics. In particular, maximum likelihood and prediction error methods represent the mainstream approaches to identification of linear dynamic systems, with a long history of theoretical and algorithmic contributions. Parallel to this, in the machine learning community alternative techniques have been developed. Until recently, there has been little contact between these two worlds. The first aim of this survey is to make accessible to the control community the key mathematical tools and concepts as well as the computational aspects underpinning these learning techniques. In particular, we focus on kernel-based regularization and its connections with reproducing kernel Hilbert spaces and Bayesian estimation of Gaussian processes. The second aim is to demonstrate that learning techniques tailored to the specific features of dynamic systems may outperform conventional parametric approaches for identification of stable linear systems.

show abstract

Section: Computational Issuesmentioning

confidence: 99%

Kernel methods in system identification, machine learning and function estimation: A survey

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Many learning problems can be formulated as convex (or even linear or semidefinite) optimizations (Bennett & Parrado-Hernández, 2006). In these problems, the data (points) act as constraints to the resulting optimization; for example, in a standard SVM formulation, there is one constraint for each point in the training set.…”

Section: Distributed Optimizationmentioning

confidence: 99%

Efficient Protocols for Distributed Classification and Optimization

Daumé

Phillips

Saha

et al. 2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In distributed learning, the goal is to perform a learning task over data distributed across multiple nodes with minimal (expensive) communication. Prior work (Daumé III et al., 2012) proposes a general model that bounds the communication required for learning classifiers while allowing for ε training error on linearly separable data adversarially distributed across nodes.In this work, we develop key improvements and extensions to this basic model. Our first result is a two-party multiplicative-weight-update based protocol that uses O(d 2 log 1/ε) words of communication to classify distributed data in arbitrary dimension d, ε-optimally. This readily extends to classification over k nodes with O(kd 2 log 1/ε) words of communication. Our proposed protocol is simple to implement and is considerably more efficient than baselines compared, as demonstrated by our empirical results.In addition, we illustrate general algorithm design paradigms for doing efficient learning over distributed data. We show how to solve fixed-dimensional and high dimensional linear programming efficiently in a distributed setting where constraints may be distributed across nodes. Since many learning problems can be viewed as convex optimization problems where constraints are generated by individual points, this models many typical distributed learning scenarios. Our techniques make use of a novel connection from multipass streaming, as well as adapting the multiplicative-weight-update framework more generally to a distributed setting. As a consequence, our methods extend to the wide range of problems solvable using these techniques.

show abstract

“…Other, algorithmically oriented, criteria have been proposed in [22]. These refer to the algorithms available to address the classification problem, and they include their scalability to handle large data sets, their performance in terms of running times and memory requirements, easiness to implement, and robustness and numerical stability.…”

Section: Performance Criteriamentioning

confidence: 99%

“…In the numerical experiments reported in [223] it is concluded that, in general, the generalization improved as the approximation gap decreased, but such improvement in generalization became rather insignificant. See also [22,211].…”

Section: Training the Svmmentioning

confidence: 99%

See 1 more Smart Citation

Supervised classification and mathematical optimization

Carrizosa

Morales

2013

Computers & Operations Research

124

View full text Add to dashboard Cite

Data Mining techniques often ask for the resolution of optimization problems. Supervised Classification, and, in particular, Support Vector Machines, can be seen as a paradigmatic instance. In this paper, some links between Mathematical Optimization methods and Supervised Classification are emphasized. It is shown that many different areas of Mathematical Optimization play a central role in off-the-shelf Supervised Classification methods. Moreover, Mathematical Optimization turns out to be extremely useful to address important issues in Classification, such as identifying relevant variables, improving the interpretability of classifiers or dealing with vagueness/noise in the data.

show abstract

Bilevel Optimization and Machine Learning

Cited by 45 publications

References 12 publications

Kernel methods in system identification, machine learning and function estimation: A survey

Kernel methods in system identification, machine learning and function estimation: A survey

Efficient Protocols for Distributed Classification and Optimization

Supervised classification and mathematical optimization

Contact Info

Product

Resources

About