Deep Reinforcement Learning for Green Security Games with Real-Time Information

Abstract

Green Security Games (GSGs) have been proposed and applied to optimize patrols conducted by law enforcement agencies in green security domains such as combating poaching, illegal logging and overfishing. However, real-time information such as footprints and agents' subsequent actions upon receiving the information, e.g., rangers following the footprints to chase the poacher, have been neglected in previous work. To fill the gap, we first propose a new game model GSG-I which augments GSGs with sequential movement and the vital element of real-time information. Second, we design a novel deep reinforcement learning-based algorithm, DeDOL, to compute a patrolling strategy that adapts to the real-time information against a best-responding attacker. DeDOL is built upon the double oracle framework and the policy-space response oracle, solving a restricted game and iteratively adding best response strategies to it through training deep Q-networks. Exploring the game structure, DeDOL uses domain-specific heuristic strategies as initial strategies and constructs several local modes for efficient and parallelized training. To our knowledge, this is the first attempt to use Deep Q-Learning for security games.

Iterated Deep Reinforcement Learning in Games

“…Specifically, we found that after a handful of rounds, the quality of learned attacker and defender reached a roughly comparable level with those learned in the same round of the heuristic-driven HADO-EGTA runs. This result contrasts with the experience of Wang et al [46], who found iterative deep RL to be ineffective for their game when starting from random strategies, but successful when seeded with hand-coded heuristics. This disparity could be due to differences in the game setting, the methods, or a combination.…”

“…PSRO generalizes what we are terming DO-EGTA primarily by introducing meta-strategy solvers (discussed further below), which support a range of approaches for generating opponents to train against in strategy generation. Recently, Wang et al [46] applied DQN as the oracle for a zero-sum dynamic form of green security game. They found that employing double oracle with DQN may take many rounds and substantial computation to converge, even for modestly sized environments.…”

“…In the security domain, Kamra et al [18] trained a DNN to learn policies for a green security game (zero-sum between a ranger and poachers) with continuous actions. Wang et al [46] used an iterated deep RL procedure, like ours a version of PSRO, to develop new strategies in a dynamic green security game. Venkatesan et al [45] demonstrated a system using reinforcement learning (without DNNs) to determine the placement of intrusion detectors to defend a computer network.…”

“…Bansal et al [4] propose opponent sampling, which trains a DNN against a randomly sampled old opponent instead of the current opponent, and appears to speed up training of competitive agents in simulated physics games (perhaps by providing a form of curriculum learning where neither training opponent can improve too quickly), as well as improving the robustness of the trained agents to unseen opponents. Wang et al [46] promote generalization by training against a weighted average of the current equilibrium opponent mixed strategy and the uniform distribution over opponent pure strategies. Foerster et al [8] introduce Learning with Opponent Learning Awareness (LOLA), a deep RL learning rule intended to stabilize MARL training, that models the opponent's EC'19 Session 6a: Game Theory policy, predicts the opponent's policy update, and updates one's own policy in anticipation of the opponent's update.…”

“…Lanctot et al [21] generalized DO for combination with deep RL in a procedure called policy space response oracle (PSRO). Wang et al [46] employ a version of PSRO to learn effective strategies in a class of green security games.…”