Model-free Adaptive Optimal Control of Episodic Fixed-horizon Manufacturing Processes Using Reinforcement Learning

Abstract

A self-learning optimal control algorithm for episodic fixed-horizon manufacturing processes with time-discrete control actions is proposed and evaluated on a simulated deep drawing process. The control model is built during consecutive process executions under optimal control via reinforcement learning, using the measured product quality as reward after each process execution. Prior model formulation, which is required by state-of-the-art algorithms from model predictive control and approximate dynamic programming, is therefore obsolete. This avoids several difficulties namely in system identification, accurate modelling, and runtime complexity, that arise when dealing with processes subject to nonlinear dynamics and stochastic influences. Instead of using pre-created process and observation models, value function-based reinforcement learning algorithms build functions of expected future reward, which are used to derive optimal process control decisions. The expectation functions are learned online, by interacting with the process. The proposed algorithm takes stochastic variations of the process conditions into account and is able to cope with partial observability. A Q-learning-based method for adaptive optimal control of partially observable episodic fixed-horizon manufac-turing processes is developed and studied. The resulting algorithm is instantiated and evaluated by applying it to a simulated stochastic optimal control problem in metal sheet deep drawing.

Multiobjective Reinforcement Learning for Reconfigurable Adaptive Optimal Control of Manufacturing Processes

“…The reward gained per episode is visualized as box-plots grouped by the position of the task-instance in the task-sequence and aggregated over the 100 experiments and over 250 episodes per box. The baseline (red box) is consisting of 100 random taskinstances, independently optimized by single-policy learning with single-objective manufacturing-process NFQ, as proposed in [8]. The plot shows a positive effect of the amount of prior knowledge to the convergence speed.…”

“…Furthermore stochastic process behavior is induced by varying friction-coefficients, randomly drawn per process-execution from a beta-distribution with α = 3, β = 15. A detailed description of the evaluation environment can be found in [8].…”

“…Instead of a single NN, time-dependentQ-models are used, which are trained backwards in time-steps. An in-depth description of the application-specific NFQ adoptions can be found in our previous work [8].…”

Deep reinforcement learning methods for structure-guided processing path optimization

“…In the context of manufacturing, reinforcement learning methods have been proposed for model-free adaptive optimization on the device level and the operational level of various manufacturing processes. Recently published work include the optimization of process control in sheet metal milling (Veeramani et al 2019), polymerization reaction systems (Ma et al 2019), laser welding (Günther et al 2016) and in deep drawing (Dornheim et al 2019). Operational optimization objects are amongst others material flow in industrial mining (Kumar et al 2020), preventive maintenance scheduling of flow line systems (Wang et al 2016) and job shop scheduling (Kuhnle et al 2020).…”

Analysis and Experimental Research of Model-Free Reinforcement Learning Method

“…Q s a π = оптимальна стратегія. Оскільки Q-функція (Q-function) робить дію явною, то оцінимо Q-значення (Q-values), використовуючи метод TD(0) [11], [13], а також визначимо стратегію (дія може бути вибрана простим відбором максимального Q-значення для поточного стану). Значення стратегії обчислюється із використанням методу TD(0), який є екземпляром більш загального класу методів TD(λ) [14].…”