Reinforcement learning based schemes to manage client activities in large distributed control systems

Laser-plasma physics has developed rapidly over the past few decades as lasers have become both more powerful and more widely available. Early experimental and numerical research in this field was dominated by single-shot experiments with limited parameter exploration. However, recent technological improvements make it possible to gather data for hundreds or thousands of different settings in both experiments and simulations. This has sparked interest in using advanced techniques from mathematics, statistics and computer science to deal with, and benefit from, big data. At the same time, sophisticated modeling techniques also provide new ways for researchers to deal effectively with situation where still only sparse data are available. This paper aims to present an overview of relevant machine learning methods with focus on applicability to laser-plasma physics and its important sub-fields of laser-plasma acceleration and inertial confinement fusion.

show abstract

“…There are many other examples for the use of RL at accelerator facilities, for example, Refs. [240][241][242][243].…”

Section: Model State Environmentmentioning

confidence: 99%

Data-driven science and machine learning methods in laser–plasma physics

Döpp

Eberle

Howard

et al. 2023

High Pow Laser Sci Eng

View full text Add to dashboard Cite

show abstract

“…Additional studies using MOGA for accelerator optimization were conducted by Li et al (2018) and Neveu et al (2019). Reinforcement learning tools have also been developed to optimize various elements of the accelerator system (Gao et al, 2019;Bruchon et al, 2020;Hirlaender and Bruchon, 2020;Kain et al, 2020;O'Shea, Bruchon, and Gaio, 2020;John et al, 2021). Additionally, there is new research on transferring the RL policy models to a field-programmable gate array to provide a low latency control response time (John et al, 2021).…”

Section: Control Optimizationmentioning

confidence: 99%

Colloquium: Machine learning in nuclear physics

et al. 2022

View full text Add to dashboard Cite

TABLE I. Table of ML methods discussed in this Colloquium with an indication of the main type of learning (S, supervised; U, unsupervised; semi-S, semisupervised). Acronym Method Description Learning type AE, VAE Autoencoders, Variational autoencoders ANN capable of learning efficient representations of the input data without any supervision U ANN Artificial neural network Models for learning defined by connected units (or nodes) and hidden layers with well-defined inputs and outputs S BED Bayesian experimental design Bayesian inference for experimental design S BM Boltzmann machine Generative ANN that can learn a probability distribution from sets of changing inputs U BMA, BMM Bayesian model averaging, Bayesian model mixing Bayesian inference applied to model selection or the combined estimation, or performed over a mixture model S BNN Bayesian neural network ANN where the parameters of the network are represented by probabilities learned by Bayesian inference S BO Bayesian optimization Optimization of functions without an a priori knowledge of functional forms. S and semi-S CNN Convolutional neural network ANN where convolution is used to reduce dimensionalities S EMB Ensemble methods and boosting Methods based on collections of decision trees as simple learners S GAN Generative adversarial network System of two ANNs where a generative network generates outputs while a discriminative network evaluates them U GP Gaussian process Collection of random variables that have a joint Gaussian distribution used in Bayesian inference Semi-S KNN k-nearest neighbors Nonparametric method where inputs consist of the k closest training examples in a dataset S KR Kernel regression Extension of linear regression methods to include nonlinear function kernels S LR Logistic regression Convex optimization method based on maximum likelihood estimate for classification problems S LSTM Long short-term memory RNN capable of learning long-term dependencies S PCA Principal component analysis Dimensionality reduction technique based on retaining the largest eigenvalues of the covariance matrix U REG Linear regression Linear algebra methods used for modeling continuous functions in terms of their explanatory variables S RL Reinforcement learning Learning achieved by trial and error of desired and undesired events Neither S nor U RNN Recurrent neural network ANN where connections between nodes allow for temporal dynamic behavior S SVM Support vector machine Convex optimization techniques with efficient ways to distinguish features in datasets S

show abstract

“…RL algorithms have been substantially improved in many aspects in the past decades, including balancing exploration and exploitation (Sutton and Barto 2018), search strategies (Lin 2015), learning behaviour (Sutton and Barto 2018), reward evaluation (Gao et al 2019). However, there is lack of application to water resources systems or hydropower systems with a few studies using traditional RL such as Opposition-based learning, Q-learning or fitted Q-iteration (Lee and Labadie 2007;Castelletti et al 2010 and.…”

Section: Introductionmentioning

confidence: 99%

“…However, there is lack of application to water resources systems or hydropower systems with a few studies using traditional RL such as Opposition-based learning, Q-learning or fitted Q-iteration (Lee and Labadie 2007;Castelletti et al 2010 and. Traditional RL uses state decision tables to map the relationship between states and actions (Lin 2015;Gao et al 2019). With an increasing number of state variables, however, the decision table approach as in the traditional RL cannot effectively handle the large number of combinations of states and actions, resulting in the curse of dimensionality problem (Mnih et al 2013;François-Lavet et al 2018).…”

Section: Introductionmentioning

confidence: 99%

Deep Reinforcement Learning for Optimal Hydropower Reservoir Operation

Meng

Guo

et al. 2021

J. Water Resour. Plann. Manage.

View full text Add to dashboard Cite

Optimal operation of hydropower reservoir systems is a classical optimization problem of high dimensionality and stochastic nature. A key challenge lies in improving the interpretability of operation strategies, i.e., the cause-effect relationship between system outputs (or actions) and contributing variables such as states and inputs. Here we report for the first time a new Deep Reinforcement Learning (DRL) framework for optimal operation of reservoir systems based on Deep Q-Networks (DQN), which provides a significant advance in understanding the performance of optimal operations. DQN combines Q-learning and two deep ANN networks and acts as the agent to interact with the reservoir system through learning its states and providing actions. Three knowledge forms of learning considering the states, actions and rewards are constructed to improve the

show abstract

Reinforcement learning based schemes to manage client activities in large distributed control systems

Cited by 4 publications

References 7 publications

Data-driven science and machine learning methods in laser–plasma physics

Data-driven science and machine learning methods in laser–plasma physics

Colloquium: Machine learning in nuclear physics

Deep Reinforcement Learning for Optimal Hydropower Reservoir Operation

Contact Info

Product

Resources

About