Optimizing Quantum Variational Circuits with Deep Reinforcement Learning

Abstract

Quantum Machine Learning (QML) is considered to be one of the most promising applications of near term quantum devices. However, the optimization of quantum machine learning models presents numerous challenges arising from the imperfections of hardware and the fundamental obstacles in navigating an exponentially scaling Hilbert space. In this work, we evaluate the potential of contemporary methods in deep reinforcement learning to augment gradient based optimization routines in quantum variational circuits. We find that reinforcement learning augmented optimizers consistently outperform gradient descent in noisy environments. All code and pretrained weights are available to replicate the results or deploy the models at github.com/lockwo/rl_qvc_opt.

An Empirical Review of Optimization Techniques for Quantum Variational Circuits

“…. e −iβ0Hm e −iγ0Hc |Ψ 0 • Excited state classification: a dataset of cluster states to classify as either containing a X gate or not, uses the mean squared error between the expected value of the Z operator on the last qubit and the target label as the loss function • Moon binary classification: a generated dataset of two classes of two dimensional datapoints (moon, circle and blob visualizations can be seen in [Lockwood, 2021]), uses the cross entropy between the target label and the the expected values of the Z operator on both qubits as the loss function • Circle binary classification: a generated dataset of two circles (one enclosed by the other) with two dimensional inputs, uses the cross entropy between the target label and the the expected value of the Z operator on first qubit as the loss function • Blobs multiclass classification: a generated dataset of a collection of blobs with two dimensional inputs, uses the categorical cross entropy between the Z expectation values on all the qubits and the labels as the loss function • Regression on the Boston Housing dataset: attempts to predict the cost of houses based on a 13 dimensional input, uses the mean squared error between the labels and the Z expectation value of the first qubit as the loss function…”

“…Shor's algorithm [Shor, 1994, Gidney andEkerå, 2021]). To this end, a number of Quantum Machine Learning (QML) routines have been proposed , Bharti et al, 2021 for supervised , Cong et al, 2019, Blank et al, 2020, unsupervised [Aïmeur et al, 2007, Dallaire-Demers and Killoran, 2018, and reinforcement learning , Lockwood and Si, 2020, 2021. Although many of these algorithms have claims of exponential scaling properties that enable them to exploit smaller hardware systems in theory, there are still a number of challenges in practice.…”

“…In the training routines of QVCs there are a number of methods employed to classically calculate the parameter updates, including traditional black block optimizers, parameter-shift based gradient optimizers [Wierichs et al, 2021], and machine learning based optimizers [Sørdal and Bergli, 2019, Lockwood, 2021. There is often little theoretical motivation for any given optimizer, although recent work has been done to analyze the training dynamics of QVCs [Liu et al, 2021b].…”

Trainability Enhancement of Parameterized Quantum Circuits via Reduced-Domain Parameter Initialization

“…Moreover, in [52,66], it was suggested that when the target Hamiltonian is a linear combination of unitaries one can circumvent the BP issue by constructing the ansatz in an iterative way. Ansatzes can also be trained layer-by-layer [97,98], or designed by classical reinforcement learning [99].…”

Quantum logic gate synthesis as a Markov decision process

“…There have been numerous previous studies in the general problem of quantum compilation, including the Solovay-Kitaev algorithm 15 , quantum Shannon decomposition 16 , approximate compilation 17,18 , as well as optimal circuit synthesis [19][20][21] . Recent applications of reinforcement learning to quantum computing include finding optimal parameters in variational quantum circuits [22][23][24] , quantum versions of reinforcement learning and related methods [25][26][27][28][29][30][31] , Bell tests 32 , as well as quantum control [33][34][35][36] , state engineering, and gate compilation [37][38][39][40][41][42] . In such studies, reinforcement learning is employed as an approximate solver of some underlying MDP.…”