Complex Momentum for Optimization in Games

Abstract

We generalize gradient descent with momentum for learning in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a dropin replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in adversarial games-like generative adversarial networks-by showing we can find better solutions with an almost identical computational cost. We also show a practical generalization to a complex-valued Adam variant, which we use to train BigGAN to better inception scores on CIFAR-10.

Training GAN with predictive centripetal acceleration

“…事实上, 在二元线性博弈中能够体现出 GDA 在 GAN 训练过程中存在的困难, 同时, 二元线性博弈可代表一些简单形式的 GAN, 如生成器和判别器均为线性函数的 Wasserstein GAN (参 见文献 [12,20,40,49]). 此外, 自 2016 年 Goodfellow [23] 分析了二维二元线性博弈上 GDA 连续时间轨 迹为圆的方程之后, 基于 GDA 提出新算法的研究几乎都在二元线性博弈上讨论了所提出算法的有效 性, 如近期文献 [7,12,13,20,22,33,40,62] 以及文献 [3,5,6,11,36,42,43,48] 和 [17,19,61] 等.…”