Towards on-sky adaptive optics control using reinforcement learning

Abstract

Context. The direct imaging of potentially habitable Exoplanets is one prime science case for the next generation of high contrast imaging instruments on ground-based extremely large telescopes. To reach this demanding science goal, the instruments are equipped with eXtreme Adaptive Optics (XAO) systems which will control thousands of actuators at a framerate of kilohertz to several kilohertz. Most of the habitable exoplanets are located at small angular separations from their host stars, where the current control laws of XAO systems leave strong residuals. Aims. Current AO control strategies like static matrix-based wavefront reconstruction and integrator control suffer from temporal delay error and are sensitive to mis-registration, i.e., to dynamic variations of the control system geometry. We aim to produce control methods that cope with these limitations, provide a significantly improved AO correction, and, therefore, reduce the residual flux in the coronagraphic point spread function (PSF). Methods. We extend previous work in Reinforcement Learning (RL) for AO. The improved method, called PO4AO, learns a dynamics model and optimizes a control neural network, called a policy. We introduce the method and study it through numerical simulations of XAO with Pyramid wavefront sensing for the 8-m and 40-m telescope aperture cases. We further implemented PO4AO and carried out experiments in a laboratory environment using Magellan Adaptive Optics eXtreme system (MagAO-X) at the Steward laboratory.Results. PO4AO provides the desired performance by improving the coronagraphic contrast in numerical simulations by factors 3-5 within the control region of DM and Pyramid WFS, both in simulation and in the laboratory. The presented method is also quick to train, i.e., on timescales of typically 5-10 seconds, and the inference time is sufficiently small (< ms) to be used in real-time control for XAO with currently available hardware even for extremely large telescopes.

MagAO-X: current status and plans for Phase II

“…24 MagAO-X can also be used for remote research and development, as demonstrated by the testing of a reinforcement learning algorithm. 25 MagAO-X is also part of the High-Contrast Adaptive Optics Testbed (HCAT) for the Giant Magellan Telescope (GMT), part of the overall program to develop segment phase sensing and control strategies for the GMT. 7 As part of HCAT MagAO-X serves as the AO system and houses phase sensing and control experiments.…”

Model-free reinforcement learning with a non-linear reconstructor for closed-loop adaptive optics control with a pyramid wavefront sensor

“…8 On the other hand, a few predictive models based on reinforcement learning (RL) have been developed. [9][10][11][12][13][14][15] In this work, we focus on closed-loop RL methods with a wavefront sensor (WFS), in which the goal is to learn the control policy, a function that maps information of the system to commands, during operation. [11][12][13][14][15] In this case, there is no distribution mismatch between training and operational data as they are the same.…”

“…[9][10][11][12][13][14][15] In this work, we focus on closed-loop RL methods with a wavefront sensor (WFS), in which the goal is to learn the control policy, a function that maps information of the system to commands, during operation. [11][12][13][14][15] In this case, there is no distribution mismatch between training and operational data as they are the same. Moreover, in contrast to most current model-based predictive controllers, there are no assumptions, and the control dynamics are learnt purely from data.…”

“…RL for AO has initially been assessed with a focus on WFS-less methods. 9,10 The current RL methods considering a WFS are mainly divided into model-based 12,15 and model-free. 11,13,14 Model-based RL methods involve building a model (usually with neural networks) of the system's dynamics and use it either to find an optimal action or to train a control policy.…”

Joint optimization of wavefront sensing and reconstruction with automatic differentiation

“…[20][21][22] A promising area of research for mitigating these nonlinearities is the use of neural networks for learning a nonlinear mapping between wavefront sensor measurements and wavefront, [23][24][25][26] or for nonlinear control. [27][28][29][30] Furthermore, the similarities between optical systems and Neural Networks have lead to studies exploiting automatic differentiation algorithms, initially developed for training NNs, for optimizing elements in the optical system 31,32 or more efficient wavefront control. 33,34 Automatic differentiation allows us to obtain gradients with respect to the free design parameters, even for complex optical systems with multiple elements and planes.…”