Factorial Kernel Dynamic Policy Programming for Vinyl Acetate Monomer Plant Model Control

Cui, Yunduan; Zhu, Lingwei; Fujisaki, Morihiro; Kanokogi, Hiroaki; Matsubara, Takamitsu

doi:10.1109/coase.2018.8560593

Cited by 14 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, there has been significant interest in deep RL methods for process control [14,15,16,17,18,19,20]. Spielberg et al [8] adapted the popular model-free deep deterministic policy gradient (DDPG) algorithm for setpoint tracking problems.…”

Section: Related Workmentioning

confidence: 99%

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Lawrence,

Forbes,

Loewen

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (RL) is an optimization-driven framework for producing control strategies for general dynamical systems without explicit reliance on process models. Good results have been reported in simulation. Here we demonstrate the challenges in implementing a state of the art deep RL algorithm on a real physical system. Aspects include the interplay between software and existing hardware; experiment design and sample efficiency; training subject to input constraints; and interpretability of the algorithm and control law. At the core of our approach is the use of a PID controller as the trainable RL policy. In addition to its simplicity, this approach has several appealing features: No additional hardware needs to be added to the control system, since a PID controller can easily be implemented through a standard programmable logic controller; the control law can easily be initialized in a "safe" region of the parameter space; and the final product-a well-tuned PID controller-has a form that practitioners can reason about and deploy with confidence.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Lawrence,

Forbes,

Loewen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…From the figure, it is seen that the pursuing team has P 0 and P 1 , and the evader is denoted as E 0 . If P 0 and P 1 can obtain the rough information Vp and Ve , the suboptimal policy θ P1 and θ P0 will be obtained based on Equation (24).…”

Section: Accelerating Fuzzy Actor-critic Learning Via Suboptimal Know...mentioning

confidence: 99%

“…From the figure, it is seen that the pursuing team has 𝑃 and 𝑃 , and the evader is denoted as 𝐸 . If 𝑃 and 𝑃 can obtain the rough information 𝑉 and 𝑉 , the suboptimal policy 𝜃 and 𝜃 will be obtained based on Equation (24). Since the suboptimal policies θ P1 and θ P0 cannot guarantee the pursuers' successful capture of the evader, the FACL was introduced here to form a new learning algorithm called the accelerating fuzzy actor-critic learning algorithm via suboptimal knowledge (SK-FACL).…”

Section: Accelerating Fuzzy Actor-critic Learning Via Suboptimal Know...mentioning

confidence: 99%

“…However, in the field of practical applications, RL is facing the challenges of low efficiency and hard convergence due to the large system state and decision-making state spaces [23]. In order to improve the practical application effect of reinforcement learning, there are factorial reinforcement learning (FRL) [24], hierarchical reinforcement learning (HRL) [25], inverse reinforcement learning (IRL) [26], deep reinforcement learning (DRL), etc. [27].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Fuzzy Actor–Critic Learning via Suboptimal Knowledge for a Multi-Agent Tracking Problem

Wang

Mao

et al. 2023

Electronics

View full text Add to dashboard Cite

Multi-agent differential games usually include tracking policies and escaping policies. To obtain the proper policies in unknown environments, agents can learn through reinforcement learning. This typically requires a large amount of interaction with the environment, which is time-consuming and inefficient. However, if one can obtain an estimated model based on some prior knowledge, the control policy can be obtained based on suboptimal knowledge. Although there exists an error between the estimated model and the environment, the suboptimal guided policy will avoid unnecessary exploration; thus, the learning process can be significantly accelerated. Facing the problem of tracking policy optimization for multiple pursuers, this study proposed a new form of fuzzy actor–critic learning algorithm based on suboptimal knowledge (SK-FACL). In the SK-FACL, the information about the environment that can be obtained is abstracted as an estimated model, and the suboptimal guided policy is calculated based on the Apollonius circle. The guided policy is combined with the fuzzy actor–critic learning algorithm, improving the learning efficiency. Considering the ground game of two pursuers and one evader, the experimental results verified the advantages of the SK-FACL in reducing tracking error, adapting model error and adapting to sudden changes made by the evader compared with pure knowledge control and the pure fuzzy actor–critic learning algorithm.

show abstract

“…Some initial studies by Hoskins and Himmelblau [8], Kaisare et al [9], Lee and Lee [10], Lee and Wong [11] in the 1990s and 2000s demonstrated the appeal of reinforcement learning and approximate dynamic programming for process control applications. More recently, there has been significant interest in deep RL methods for process control [12,13,14,15,16,17,18].…”

Section: Related Workmentioning

confidence: 99%

Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach

McClement¹,

Lawrence²,

Backström³

et al. 2022

Preprint

View full text Add to dashboard Cite

Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as the system gain or time constant, yet efficiently controls novel systems in a completely model-free fashion. Our meta-RL agent has a recurrent structure that accumulates "context" for its current dynamics through a hidden state variable. This end-to-end architecture enables the agent to automatically adapt to changes in the process dynamics. Moreover, the same agent can be deployed on systems with previously unseen nonlinearities and timescales. In tests reported here, the meta-RL agent was trained entirely offline, yet produced excellent results in novel settings. A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with novel environments. To illustrate the approach, we take the actions proposed by the meta-RL agent to be changes to gains of a proportional-integral controller, resulting in a generalized, adaptive, closed-loop tuning strategy. Meta-learning is a promising approach for constructing sample-efficient intelligent controllers.

show abstract

Factorial Kernel Dynamic Policy Programming for Vinyl Acetate Monomer Plant Model Control

Cited by 14 publications

References 14 publications

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Accelerating Fuzzy Actor–Critic Learning via Suboptimal Knowledge for a Multi-Agent Tracking Problem

Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach

Contact Info

Product

Resources

About