Iterated Deep Reinforcement Learning in Games

Abstract

Deep reinforcement learning (RL) is a powerful method for generating policies in complex environments, and recent breakthroughs in game-playing have leveraged deep RL as part of an iterative multiagent search process. We build on such developments and present an approach that learns progressively better mixed strategies in complex dynamic games of imperfect information, through iterated use of empirical game-theoretic analysis (EGTA) with deep RL policies. We apply the approach to a challenging cybersecurity game defined over attack graphs. Iterating deep RL with EGTA to convergence over dozens of rounds, we generate mixed strategies far stronger than earlier published heuristic strategies for this game. We further refine the strategy-exploration process, by fine-tuning in a training environment that includes out-of-equilibrium but recently seen opponents. Experiments suggest this history-aware approach yields strategies with lower regret at each stage of training. CCS Concepts: • Theory of computation → Exact and approximate computation of equilibria; Quality of equilibria; • Computing methodologies → Multi-agent reinforcement learning.

PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML

“…The input image size is 480 × 480, and the maximum sequence length is 500. Synchronized BN [4] and Ranger optimizer [5] are apply in this experiment, and the initial learning rate of optimizer is 0.001 with step learning rate decay.…”

1st Place Solution for ICDAR 2021 Competition on Mathematical Formula Detection

“…Ranger [16] is a synergistic optimizer combining RAdam (Rectified Adam) [22], LookAhead [23], and GC (gradient centralization) [24].…”

“…In addition, we leveraged the feature pyramid network (FPN) [11], the large input resolution, and the deformable convolutional network (DCN) [12] because these techniques could benefit large formula detection or small formula detection, or both. Finally, some other tricks, such as ResNeSt [13], SyncBN [14], the large batch size and the weighted box fusion (WBF) [15], Ranger [16] optimizer, were adopted in our solution.…”

PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex

“…Ranger Optimizer. Ranger [3] integrates RAdam (Rectified Adam) [4], LookAhead [5], and GC (gradient centralization) [6] into one optimizer. LookAhead can be considered as an extension of Stochastic Weight Averaging (SWA) [7] in the training stage.…”

“…The maximum sequence length is 500. In default, Synchronized BN [10] and Ranger optimizer [3] are used in our experiments. The initial learning rate of optimizer is 0.001 with step learning rate decay.…”