Living on the Edge: A Geometric Theory of Phase Transitions in Convex Optimization

Amelunxen, Dennis; Lotz, Martin; McCoy, Michael B.; Tropp, Joel A.

doi:10.21236/ada591124

Cited by 73 publications

(270 citation statements)

References 57 publications

Supporting

Mentioning

255

Contrasting

Unclassified

Order By: Relevance

“…[66]) and a large number of sharp transitions in random geometry that are also called phase transitions (see e.g. [57], or the more theoretical paper [67]). …”

Section: Summary and Discussionmentioning

confidence: 99%

Replica approach to mean-variance portfolio optimization

2016

View full text Add to dashboard Cite

We consider the problem of mean-variance portfolio optimization for a generic covariance matrix subject to the budget constraint and the constraint for the expected return, with the application of the replica method borrowed from the statistical physics of disordered systems. We find that the replica symmetry of the solution does not need to be assumed, but emerges as the unique solution of the optimization problem. We also check the stability of this solution and find that the eigenvalues of the Hessian are positive for r = N/T < 1, where N is the dimension of the portfolio and T the length of the time series used to estimate the covariance matrix. At the critical point r = 1 a phase transition is taking place. The out of sample estimation error blows up at this point as 1/(1 − r), independently of the covariance matrix or the expected return, displaying the universality not only of the critical exponent, but also the critical point. As a conspicuous illustration of the dangers of in-sample estimates, the optimal in-sample variance is found to vanish at the critical point inversely proportional to the divergent estimation error.

show abstract

“…[66]) and a large number of sharp transitions in random geometry that are also called phase transitions (see e.g. [57], or the more theoretical paper [67]). …”

Section: Summary and Discussionmentioning

confidence: 99%

Replica approach to mean-variance portfolio optimization

2016

View full text Add to dashboard Cite

show abstract

“…Our realization of the tradeoff relies on recent work in convex geometry that allows for precise analysis of statistical risk. In particular, we recognize the work done by Amelunxen et al [3] to identify phase transitions in regularized linear inverse problems and the extension to noisy problems by Oymak and Hassibi [4]. While we illustrate our smoothing approach using this single class of problems, we believe that many other examples exist.…”

mentioning

confidence: 81%

Designing Statistical Estimators That Balance Sample Size, Risk, and Computational Cost

Bruer

Tropp

Cevher

et al. 2015

IEEE J. Sel. Top. Signal Process.

Self Cite

View full text Add to dashboard Cite

Abstract-This paper proposes a tradeoff between computational time, sample complexity, and statistical accuracy that applies to statistical estimators based on convex optimization. When we have a large amount of data, we can exploit excess samples to decrease statistical risk, to decrease computational cost, or to trade off between the two. We propose to achieve this tradeoff by varying the amount of smoothing applied to the optimization problem. This work uses regularized linear regression as a case study to argue for the existence of this tradeoff both theoretically and experimentally. We also apply our method to describe a tradeoff in an image interpolation problem.Index Terms-Smoothing methods, statistical estimation, convex optimization, regularized regression, image interpolation, resource tradeoffs I. MOTIVATION M ASSIVE DATA presents an obvious challenge to statistical algorithms. We expect that the computational effort needed to process a data set increases with its size. The amount of computational power available, however, is growing slowly relative to sample sizes. As a consequence, large-scale problems of practical interest require increasingly more time to solve. This creates a demand for new algorithms that offer better performance when presented with large data sets.While it seems natural that larger problems require more effort to solve, Shalev-Shwartz and Srebro [1] showed that their algorithm for learning a support vector classifier actually becomes faster as the amount of training data increases. This and more recent works support an emerging viewpoint that treats data as a computational resource. That is, we should be able to exploit additional data to improve the performance of statistical algorithms.We consider statistical problems solved through convex optimization and propose the following approach: We can smooth statistical optimization problems more and more aggressively as the amount of available data increases. By controlling the amount of smoothing, we can exploit the additional data to decrease statistical risk, decrease computational cost, or trade off between the two. Our prior work [2] examined a similar time-data tradeoff achieved by applying a dual-smoothing method to (noiseless) regularized linear inverse problems. This paper generalizes those results, allowing for noisy measurements. The result is a tradeoff in computational time, sample size, and statistical accuracy.We use regularized linear regression problems as a specific example to illustrate our principle. We provide theoretical and numerical evidence that supports the existence of a time-data tradeoff achievable through aggressive smoothing of convex optimization problems in the dual domain. Our realization of the tradeoff relies on recent work in convex geometry that allows for precise analysis of statistical risk. In particular, we recognize the work done by Amelunxen et al.[3] to identify phase transitions in regularized linear inverse problems and the extension to noisy problems by Oymak and Hassibi [4]. While we...

show abstract

“…Random matrices play a central role in the design and analysis of measurement procedures. For example, see [66,36,9,185].…”

Section: Subsampling Of Datamentioning

confidence: 99%

“…A common model for this problem assumes that the signals are randomly oriented with respect to each other, which means that it is usually possible to discriminate the underlying structures. Random orthogonal matrices arise in the analysis of estimation techniques for this problem [125,9,126].…”

Section: Modelingmentioning

confidence: 99%