Machine learning techniques for autonomous agents in military simulations — Multum in parvo

Roessingh, Jan Joris; Toubman, Armon; Oijen, Joost van; Poppinga, Gerald; Løvlid, Rikke Amilde; Hou, Ming; Luotsinen, Linus J.

doi:10.1109/smc.2017.8123163

Cited by 18 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this toy-example (discussed for clarification purposes), considering 9 RBFs together with localized observation vectors of size 12 for the predator and 10 for the preys, the mean vector associated with the predator and the preys are of dimensions 9 × 12 and 9 × 10, respectively. Consequently, for this Predator-Prey scenario, µ, which is initialized randomly contains three agents with random values with the mean size ((9, 12), (9, 10), (9,10)) and the covariance, Σ = (I 12 , I 10 , I 10 ) where I 12 and I 10 are the identity…”

Section: Experimental Assumptionsmentioning

confidence: 99%

“…Generally speaking, the main underlying objective is learning (via trial and error) from previous interactions of an autonomous agent and its surrounding environment. The optimal control (action) policy can be obtained via RL algorithms through the feedback that environment provides to the agent after each of its actions [3][4][5][6]8,9]. Policy optimality can be reached via such an approach with the goal of increasing the reward over time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Salimibeni¹,

Mohammadi²,

Malekzadeh³

et al. 2021

Preprint

View full text Add to dashboard Cite

Development of distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted an increasing surge of interest lately mainly due to the recent advancements of Deep Neural Networks (DNNs). Complex cooperative, competitive or mixed behavior among the agents in the multi-agent environments, make them more appealing to real world scenarios. Generally speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While DNN-based solutions perform utterly well when a single agent is involved, such methods fail to fully generalize to the complexities of MARL problems. In other words, although recently developed approaches based on DNNs for multi-agent environments have achieved superior performance, they are still prone to overfiting, high sensitivity to parameter selection, and sample inefficiency. In this paper, an adaptive Kalman filter-based framework is introduced as an efficient alternative to address the aforementioned problems. More specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. Intuitively speaking, the main objective is to capitalize on unique characteristics of Kalman Filtering (KF) such as uncertainty modeling and online second order learning. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. By leveraging the KTD framework, SR learning procedure is modeled into a filtering problem, where Radial Basis Function (RBF) estimators are used to encode the continuous space into feature vectors. On the other hand, for learning localized reward functions, we resort to Multiple Model Adaptive Estimation (MMAE), as a remedy to deal with the lack of prior knowledge on the observation noise covariance and observation mapping function. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks. In these experiments, different number of agents in cooperative, competitive and mixed (cooperative-competitive) scenarios are utilized. The experimental results illustrate superior performance of the proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.

show abstract

Section: Experimental Assumptionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Salimibeni¹,

Mohammadi²,

Malekzadeh³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, virtual humans are not new. They have been used in many domains, including advertisements as well as medical practice [1,2], healthcare [3,4], education [5,6], entertainment [7,8], and the military [9,10], interacting with the user and acting on the environment to provide a positive influence, such as behavioral change of the human counterpart [11]. The emphasis on interactivity with the user brought the term virtual agent, which utilizes verbal (e.g., conversation) and nonverbal communication (e.g., facial expressions and behavioral gestures) channels to learn, adapt, and assist the human.…”

Section: Introductionmentioning

confidence: 99%

Empathic Responses of Behavioral-Synchronization in Human-Agent Interaction

Park¹,

Park²,

Whang³

2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Artificial entities, such as virtual agents, have become more pervasive. Their long-term presence among humans requires the virtual agent's ability to express appropriate emotions to elicit the necessary empathy from the users. Affective empathy involves behavioral mimicry, a synchronized co-movement between dyadic pairs. However, the characteristics of such synchrony between humans and virtual agents remain unclear in empathic interactions. Our study evaluates the participant's behavioral synchronization when a virtual agent exhibits an emotional expression congruent with the emotional context through facial expressions, behavioral gestures, and voice. Participants viewed an emotion-eliciting video stimulus (negative or positive) with a virtual agent. The participants then conversed with the virtual agent about the video, such as how the participant felt about the content. The virtual agent expressed emotions congruent with the video or neutral emotion during the dialog. The participants' facial expressions, such as the facial expressive intensity and facial muscle movement, were measured during the dialog using a camera. The results showed the participants' significant behavioral synchronization (i.e., cosine similarity ≥ .05) in both the negative and positive emotion conditions, evident in the participant's facial mimicry with the virtual agent. Additionally, the participants' facial expressions, both movement and intensity, were significantly stronger in the emotional virtual agent than in the neutral virtual agent. In particular, we found that the facial muscle intensity of AU45 (Blink) is an effective index to assess the participant's synchronization that differs by the individual's empathic capability (low, mid, high). Based on the results, we suggest an appraisal criterion to provide empirical conditions to validate empathic interaction based on the facial expression measures.

show abstract

“…Generally speaking, the main underlying objective is learning (via trial and error) from previous interactions of an autonomous agent and its surrounding environment. The optimal control (action) policy can be obtained via RL algorithms through the feedback that environment provides to the agent after each of its actions [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ]. Policy optimality can be reached via such an approach with the goal of increasing the reward over time.…”

Section: Introductionmentioning

confidence: 99%

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Salimibeni

Mohammadi

Malekzadeh

et al. 2022

Sensors

View full text Add to dashboard Cite

Development of distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted an increasing surge of interest lately. Generally speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While Deep Neural Network (DNN)-based solutions perform well, they are still prone to overfitting, high sensitivity to parameter selection, and sample inefficiency. In this paper, an adaptive Kalman Filter (KF)-based framework is introduced as an efficient alternative to address the aforementioned problems by capitalizing on unique characteristics of KF such as uncertainty modeling and online second order learning. More specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks. In these experiments, different number of agents in cooperative, competitive, and mixed (cooperative-competitive) scenarios are utilized. The experimental results illustrate superior performance of the proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.

show abstract

Machine learning techniques for autonomous agents in military simulations — Multum in parvo

Cited by 18 publications

References 15 publications

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Empathic Responses of Behavioral-Synchronization in Human-Agent Interaction

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Contact Info

Product

Resources

About