On-Off Adversarially Robust Q-Learning

Sahoo, Prachi Pratyusha; Vamvoudakis, Kyriakos G.

doi:10.1109/lcsys.2020.2979572

Cited by 8 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other research efforts over the years have focused on addressing sequential decision-making problems under uncertainty either with direct or indirect RL methods including robust learning-based approaches in applications related to quadrotor safety and steady-state stability [64], [65], learningbased model predictive control [66]- [68] with applications on autonomous racing cars [69], real-time learning [70]- [72] of powertrain operation of vehicles with respect to the driver's driving style [73], [74], learning for planning of autonomous vehicles [75], learning for traffic control in simulation [76]- [78] in conjunction with transfer of learned policies from simulation to a scaled environment [79], [80], decentralized learning for stochastic games [81], learning for optimal social routing [82] and congestion games [83], and learning for enhanced security against replay attacks in CPS [84], [85].…”

Section: B Related Workmentioning

confidence: 99%

Separation of Learning and Control for Cyber-Physical Systems

Malikopoulos¹

2021

Preprint

View full text Add to dashboard Cite

Most cyber-physical systems (CPS) encounter a large volume of data with a dynamic nature which is added to the system gradually in real time and not altogether in advance. Therefore, neither traditional supervised (or unsupervised) learning nor typical model-based control approaches can effectively facilitate optimal solutions with performance guarantees. In this article, we provide a theoretical framework that yields optimal control strategies at the intersection of control theory and learning. In the proposed framework, we use the actual CPS, i.e., the "true" CPS that we seek to optimally control online, in parallel with a model of the CPS that we have available. We institute an information state which is the conditional joint probability distribution of the states of the model and the actual CPS given all data available up until each instant of time. We use this information state along with the CPS model to derive offline separated control strategies. Since the strategies are derived offline, the state of the actual CPS is not known, i.e., the model cannot capture the dynamics of the actual CPS due to the complexity of the system, and thus the optimal strategy of the model is parameterized with respect to the state of the actual CPS. However, the control strategy and the process of estimating the information state are separated. Therefore, we can learn the information state of the system online while we operate the model and the actual CPS. We show that after the information state becomes known online through learning, the separated control strategy of the model derived offline is optimal for the actual CPS. We illustrate the proposed framework in a dynamic system consisting of two subsystems with a delayed sharing information structure.

show abstract

Section: B Related Workmentioning

confidence: 99%

Separation of Learning and Control for Cyber-Physical Systems

Malikopoulos¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There have also been research efforts on developing learning approaches using Bayesian analysis to address such problems [4]. Other approaches over the years have focused on direct or indirect RL methods including robust learning-based [5], [6], learning-based model predictive control [7]- [9] on autonomous racing cars [10], real-time learning [11], [12] of powertain systems with respect to the driver's driving style [13], [14], learning for traffic control [15] for transferring optimal policies [16], [17], decentralized learning for stochastic games [18], learning for optimal social routing [19] and congestion games [20], and learning for enhanced security against replay attacks in cyber-physical systems [21], [22].…”

Section: Introductionmentioning

confidence: 99%

Separation of learning and control for cyber–physical systems

Malikopoulos

2023

Automatica

View full text Add to dashboard Cite

“…For example, Zhu [2019, 2021] investigate the impact of manipulating Q-learners through the falsification of cost signals and its effect on the algorithm's convergence. Correspondingly, robust variants of the Q-learning against such attacks have been studied extensively [Sahoo and Vamvoudakis, 2020, Nisioti et al, 2021, Wang et al, 2020.…”

Section: Introductionmentioning

confidence: 99%

Emotional Eater Questionnaire--Turkish Version

Arslantaş¹,

Dereboy²,

Yüksel³

et al. 2020

PsycTESTS Dataset

View full text Add to dashboard Cite

In this paper, we explore the susceptibility of the Q-learning algorithm (a classical and widely used reinforcement learning method) to strategic manipulation of sophisticated opponents in games. We quantify how much a strategically sophisticated agent can exploit a naive Q-learner if she knows the opponent's Q-learning algorithm. To this end, we formulate the strategic actor's problem as a Markov decision process (with a continuum state space encompassing all possible Q-values) as if the Q-learning algorithm is the underlying dynamical system. We also present a quantization-based approximation scheme to tackle the continuum state space and analyze its performance both analytically and numerically.

show abstract

On-Off Adversarially Robust Q-Learning

Cited by 8 publications

References 22 publications

Separation of Learning and Control for Cyber-Physical Systems

Separation of Learning and Control for Cyber-Physical Systems

Separation of learning and control for cyber–physical systems

Emotional Eater Questionnaire--Turkish Version

Contact Info

Product

Resources

About