Optimization and Machine Learning

Abstract

Optimization and machine learning are both extremely active research topics. In this thesis, we explore problems at the intersection of the two fields. In particular, we will develop two main ideas.First, optimization can be used to improve machine learning. We illustrate this idea by considering computer vision tasks that are modelled with dense conditional random fields. Existing solvers for these models are either slow or inaccurate. We show that, by introducing a specialized solver based on proximal minimization and fast filtering, these models can be solved both quickly and accurately. Similarly, we introduce a specialized linear programming solver for block bounded problems, a common class of problems encountered in machine learning. This solver is efficient, easy to tune and simple to integrate inside larger machine learning algorithms.Second, machine learning can be used to improve optimization, in particular for NP-hard problems. For problems solved by using hand-tuned heuristics, machine learning can be used to discover and improve these heuristics. We show that, for the problem of super-optimization, a better heuristic to explore the space of programs can be learnt using reinforcement learning. For problems where no such heuristics exist, machine learning can be used to get an approximate solution of the original problem. We use this idea to tackle the problem of program synthesis by reformulating it as the problem of learning a program that performs the required task. We introduce a new differentiable formulation of the execution and show that the fastest programs can be recovered for simple tasks.