Convex Optimization, Shape Constraints, Compound Decisions, and Empirical Bayes Rules

Abstract

Abstract. Estimation of mixture densities for the classical Gaussian compound decision problem and their associated (empirical) Bayes rules is considered from two new perspectives. The first, motivated by Brown and Greenshtein (2009), introduces a nonparametric maximum likelihood estimator of the mixture density subject to a monotonicity constraint on the resulting Bayes rule. The second, motivated by Jiang and Zhang (2009), proposes a new approach to computing the Kiefer-Wolfowitz nonparametric maximum likelihood estimator for mixtures. In contrast to prior methods for these problems, our new approaches are cast as convex optimization problems that can be efficiently solved by modern interior point methods. In particular, we show that the reformulation of the Kiefer-Wolfowitz estimator as a convex optimization problem reduces the computational effort by several orders of magnitude for typical problems, by comparison to prior EM-algorithm based methods, and thus greatly expands the practical applicability of the resulting methods. Our new procedures are compared with several existing empirical Bayes methods in simulations employing the well-established design of Johnstone and Silverman (2004). Some further comparisons are made based on prediction of baseball batting averages. A Bernoulli mixture application is briefly considered in the penultimate section.

Unobserved heterogeneity in income dynamics: an empirical Bayes perspective

“…The latter failing can be addressed by a further monotonization step, or by a penalization approach as suggested in Koenker and Mizera (2014). However, a more direct approach is possible via the Kiefer-Wolfowitz non-parametric maximum likelihood estimator (KWMLE) for the mixture model.…”

Adaptive Estimation of Regression Parameters for the Gaussian Scale Mixture Model

“…Implementations of all the procedures described here are available in the R package REBayes, Koenker (2012). For further details on computational aspects see Koenker and Mizera (2014).…”

REBayes: An R Package for Empirical Bayes Mixture Methods

“…The main drawback of this proposal was the painfully slow convergence of the fixed point iteration of the EM algorithm used to compute the NPMLE. Koenker and Mizera (2014b), observing that the discretization suggested by Jiang and Zhang (2009) produced a convenient, finite dimensional convex optimization problem showed that the NPMLE could be implemented much more efficiently by standard interior point methods.…”

“…Mixture models have played a central role in this revival, and this has sparked renewed interest in the Kiefer and Wolfowitz (1956) nonparametric maximum likelihood estimator (NPMLE) for mixtures. Relatively recent developments in convex optimization have dramatically improved computational methods for the Kiefer-Wolfowitz NPMLE, as described in Koenker and Mizera (2014b). To make these methods accessible to the research community we have developed an R package REBayes (Koenker 2017) that incorporates a wide variety of nonparametric mixture models and provides Kiefer-Wolfowitz procedures for each of them.…”

“…Recent developments in convex optimization, as noted by Koenker and Mizera (2014b), have led to dramatic improvements in computational methods for such models. In this vignette we describe an implementation contained in the R package REBayes with applications to a wide variety of mixture settings: Gaussian location and scale, Poisson and binomial mixtures for discrete data, Weibull and Gompertz models for survival data, and several Gaussian models intended for longitudinal data.…”