Modeling behavior of Computer Generated Forces with Machine Learning Techniques, the NATO Task Group approach

Toubman, Armon; Roessingh, Jan Joris; Oijen, Joost van; Løvlid, Rikke Amilde; Hou, Ming; Meyer, Christophe; Luotsinen, Linus J.; Roel, Rijken; Harris, Jack; Turčaník, Michal

doi:10.1109/smc.2016.7844517

Cited by 10 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although there is a study of training two combatants CGF with a supervised learning [9], supervised and unsupervised learnings are not widely used, as they require a large amount of data. In addition, it is difficult to judge whether a certain combat activity is right or wrong by using those learning methods [10]. On the other hand, reinforcement learning has the advantage in that it does not require correct answers, as agents build data by repeating episodes in the environment and improve behaviors in correct directions through rewards.…”

Section: B Cgf Automation Methodology and Reinforcement Learningmentioning

confidence: 99%

Experimental and Computational Study on the Ground Forces CGF Automation of Wargame Models Using Reinforcement Learning

et al. 2022

View full text Add to dashboard Cite

Wargame is an important tool that enables training units to develop various strategies by allowing them to experience unexpected situations. There are three methodologies that determine the behavior of the Computer Generated Forces(CGF) in wargame-rule-based, agent-based, and learningbased methodologies. The military determines the behaviors of the CGF mainly based on the rules because a doctrine and an operation plan are well established. However, the advent of intelligent weapons and the accompanying changes in tactics will make it difficult to expect an environment and situations of the future battlefield. Therefore, we studied the automation of CGF through reinforcement learning in order to give unexpected situations, so that the training unit would be able to establish various strategies and tactics through the wargame model. Based on the combat functions of the ground forces, we configured multiple environments that the ground forces CGFs will learn in. First, infantry and artillery CGFs learned in the close combat environment, which is the basis of ground forces combat. Second, the trainee CGF learned in the context of military training. Third, the drone CGF learned how to reconnaissance and attack in a multidrone environment, and finally, the combat service support CGF learned under the mission of supplying ammunition. As a result, we confirmed that the reinforcement learning methodology is applicable to CGF through these experiments.

show abstract

Section: B Cgf Automation Methodology and Reinforcement Learningmentioning

confidence: 99%

Experimental and Computational Study on the Ground Forces CGF Automation of Wargame Models Using Reinforcement Learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Generally speaking, the main underlying objective is learning (via trial and error) from previous interactions of an autonomous agent and its surrounding environment. The optimal control (action) policy can be obtained via RL algorithms through the feedback that environment provides to the agent after each of its actions [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ]. Policy optimality can be reached via such an approach with the goal of increasing the reward over time.…”

Section: Introductionmentioning

confidence: 99%

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Salimibeni

Mohammadi

Malekzadeh

et al. 2022

Sensors

View full text Add to dashboard Cite

Development of distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted an increasing surge of interest lately. Generally speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While Deep Neural Network (DNN)-based solutions perform well, they are still prone to overfitting, high sensitivity to parameter selection, and sample inefficiency. In this paper, an adaptive Kalman Filter (KF)-based framework is introduced as an efficient alternative to address the aforementioned problems by capitalizing on unique characteristics of KF such as uncertainty modeling and online second order learning. More specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks. In these experiments, different number of agents in cooperative, competitive, and mixed (cooperative-competitive) scenarios are utilized. The experimental results illustrate superior performance of the proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.

show abstract

“…Generally speaking, the main underlying objective is learning (via trial and error) from previous interactions of an autonomous agent and its surrounding environment. The optimal control (action) policy can be obtained via RL algorithms through the feedback that environment provides to the agent after each of its actions [3][4][5][6]8,9]. Policy optimality can be reached via such an approach with the goal of increasing the reward over time.…”

Section: Introductionmentioning

confidence: 99%

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Salimibeni¹,

Mohammadi²,

Malekzadeh³

et al. 2021

Preprint

View full text Add to dashboard Cite

Development of distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted an increasing surge of interest lately mainly due to the recent advancements of Deep Neural Networks (DNNs). Complex cooperative, competitive or mixed behavior among the agents in the multi-agent environments, make them more appealing to real world scenarios. Generally speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While DNN-based solutions perform utterly well when a single agent is involved, such methods fail to fully generalize to the complexities of MARL problems. In other words, although recently developed approaches based on DNNs for multi-agent environments have achieved superior performance, they are still prone to overfiting, high sensitivity to parameter selection, and sample inefficiency. In this paper, an adaptive Kalman filter-based framework is introduced as an efficient alternative to address the aforementioned problems. More specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. Intuitively speaking, the main objective is to capitalize on unique characteristics of Kalman Filtering (KF) such as uncertainty modeling and online second order learning. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. By leveraging the KTD framework, SR learning procedure is modeled into a filtering problem, where Radial Basis Function (RBF) estimators are used to encode the continuous space into feature vectors. On the other hand, for learning localized reward functions, we resort to Multiple Model Adaptive Estimation (MMAE), as a remedy to deal with the lack of prior knowledge on the observation noise covariance and observation mapping function. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks. In these experiments, different number of agents in cooperative, competitive and mixed (cooperative-competitive) scenarios are utilized. The experimental results illustrate superior performance of the proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.

show abstract

Modeling behavior of Computer Generated Forces with Machine Learning Techniques, the NATO Task Group approach

Cited by 10 publications

References 5 publications

Experimental and Computational Study on the Ground Forces CGF Automation of Wargame Models Using Reinforcement Learning

Experimental and Computational Study on the Ground Forces CGF Automation of Wargame Models Using Reinforcement Learning

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Contact Info

Product

Resources

About