Variational Quantum Circuits for Deep Reinforcement Learning

Abstract

The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep Q-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.

Comparing quantum hybrid reinforcement learning to classical methods

“…The idea from Section 2.2 was extended to VQCs as the approximator for the action-value function in Chen et al (2019), which is called variational quantum deep Q-learning (VQ-DQN). The architecture from Fig.…”

“…The gradient of the VQC with respect to θ can be computed using the Parameter-Shift Rule . The values indicated with were taken from Chen et al (2019) Thus, the DQN algorithm can be generalized to a hybrid quantum reinforcement learning method by replacing the NN approximator with a VQC. Within their publication, Chen et al used a non-pure VQC to approximate the action-value function.…”

“…Similar to NNs on classical computers, variational quantum circuits (VQCs) started to generate interest as a quantum hybrid approximation architecture. Chen et al introduced VQCs as a function approximator in RL (Chen et al 2019). They modified the well-established DQN (Mnih et al 2015) algorithm by utilizing a VQC embedded in a NN as an approximator.…”

TensorFlow Quantum: Impacts of Quantum State Preparation on Quantum Machine Learning Performance

“…Q UANTUM Machine Learning (QML) is an interdisciplinary field where Quantum Computing (QC) and Machine Learning (ML) converge. Interest in QML over the last couple of years has grown largely due to the advances in hardware implementations of quantum devices known as Noisy Intermediate Scale Quantum (NISQ) devices [1], [2]. The goal of this rising field is to describe learning models that apply the benefits of computing on quantum devices so that operations in machine learning can be performed [3] and potentially improved.…”

“…The qubit is the quantum dual of the binary bit and is represented as |ψ (read state psi or ket psi). An example of a qubit is defined in (1), where α n corresponds to the probability amplitudes constrained by (2), |0 . .…”