Min-max saddle point games appear in a wide range of applications in machine leaning and signal processing. Despite their wide applicability, theoretical studies are mostly limited to the special convex-concave structure. While some recent works generalized these results to special smooth non-convex cases, our understanding of non-smooth scenarios is still limited. In this work, we study special form of non-smooth min-max games when the objective function is (strongly) convex with respect to one of the player's decision variable. We show that a simple multi-step proximal gradient descent-ascent algorithm converges to -first-order Nash equilibrium of the min-max game with the number of gradient evaluations being polynomial in 1/ . We will also show that our notion of stationarity is stronger than existing ones in the literature. Finally, we evaluate the performance of the proposed algorithm through adversarial attack on a LASSO estimator.Keywords-Non-convex min-max games, First-order Nash equilibria, Proximal gradient descent ascent IntroductionNon-convex min-max saddle point games appear in a wide range of applications such as training Generative Adversarial Networks [1,2,3,4], fair statistical inference [5,6,7], and training robust neural networks and systems [8,9,10]. In such a game, the goal is to solve the optimization problem of the formwhich can be considered as a two player game where one player aims at increasing the objective, while the other tries to minimize the objective. Using game theoretic point of view, we may aim for finding Nash equilibria [11] in which no player can do better off by unilaterally changing its strategy. Unfortunately, finding/checking such Nash equilibria is hard in general [12] for non-convex objective functions. Moreover, such Nash equilibria might not even exist. Therefore, many works focus on special cases such as convex-concave problems where f (θ, .) is concave for any given θ and f (., α) is convex for any given α. Under this assumption, different algorithms such as optimistic mirror descent [13,14,15,16], Frank-Wolfe algorithm [17,18] and Primal-Dual method [19] have been studied.In the general non-convex settings, [20] considers the weakly convex-concave case and proposes a primal-dual based approach for finding approximate stationary solutions. More recently, the research works [21,22,23,24] * This arXiv submission includes the details of the proofs for the paper accepted for publication in the proceeding of the 45 th International Conference on Acoustics, Speech, and Signal Processing (ICASSP). arXiv:2003.08093v1 [math.OC] 18 Mar 2020examine the min-max problem in non-convex-(strongly)-concave cases and proposed first-order algorithms for solving them. Some of the results have been accelerated in the "Moreau envelope regime" by the recent interesting work [25]. This work first starts by studying the problem in smooth strongly convex-concave and convex-concave settings, and proposes an algorithm based on the combination of Mirror-Prox [26] and Nesterov's accelerated gra...

Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging nonconvex problems. While the Expectation-Maximization (EM) algorithm is the most popular approach for solving these non-convex problems, the behavior of this algorithm is not well understood. In this work, we focus on the case of mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple equally weighted mixture of two single dimensional Laplacian distributions and show that every local optimum of the population maximum likelihood estimation problem is globally optimal. Then, we prove that the EM algorithm converges to the ground truth parameters almost surely with random initialization. Our result extends the existing results for Gaussian distribution to Laplacian distribution. Then we numerically study the behavior of mixture models with more than two components. Motivated by our extensive numerical experiments, we propose a novel stochastic method for estimating the mean of components of a mixture model. Our numerical experiments show that our algorithm outperforms the Naïve EM algorithm in almost all scenarios.Index Terms-Finite mixture model, Gaussian/Laplacian mixture model, EM algorithm, non-convex optimization

Adaptive momentum methods have recently attracted a lot of attention for training of deep neural networks. They use an exponential moving average of past gradients of the objective function to update both search directions and learning rates. However, these methods are not suited for solving min-max optimization problems that arise in training generative adversarial networks. In this paper, we propose an adaptive momentum min-max algorithm that generalizes adaptive momentum methods to the non-convex min-max regime. Further, we establish non-asymptotic rates of convergence for it when used in a reasonably broad class of non-convex min-max optimization problems. Experimental results illustrate its superior performance vis-a-vis benchmark methods for solving such problems.