Magnetic control of tokamak plasmas through deep reinforcement learning

Abstract

Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable1,2, including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and ‘snowflake’ configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained ‘droplets’ on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.

Unentangled quantum reinforcement learning agents in the OpenAI Gym

“…First, the expectation value of the measurements is obtained from Eq. (3) denoted by y iq = f(x q i , θ q i ) = 0 U † (x q i , θ q i ) Z q U(x q i , θ q i ) 0 , (5) where q = 1, . .…”

“…Classical reinforcement learning (RL) [1] has generated excellent results in different regions [2][3][4][5][6][7]. During the past decade, RL has been broadly applied to master Go [2], design chips [7], play the game for StarCraft and Gran Turismo [3,4], improve the nuclear fusion problem [5], and solve the problem of protein folding [6]. Despite the remarkable achievements, most RL techniques fail to balance the tradeoff between exploitation and exploration [8].…”

Reinforcement Learning in Practice: Opportunities and Challenges

“…See Section 4.1 for more discussion. Degrave et al (2022) report an extraordinary attainment applying deep RL to nuclear fusion, promising for sustainable energy. RL has been applied to chemical retrosynthesis (Segler et al, 2018) and drug discovery (Popova et al, 2018;Zhavoronkov et al, 2019).…”

Distributional Reinforcement Learning for Scheduling of Chemical Production Processes

“…Although RL has been demonstrated for sequential decision making in a number of case studies [21,22,23,24], its application to physical production systems has been relatively limited. For example, [25] applied deep Q networks to optimize a flexible jobshop (i.e.…”