Learning Unitary Operators with Help From u(n)

Abstract

A major challenge in the training of recurrent neural networks is the so-called vanishing or exploding gradient problem. The use of a norm-preserving transition operator can address this issue, but parametrization is challenging. In this work we focus on unitary operators and describe a parametrization using the Lie algebra u(n) associated with the Lie group U(n) of n × n unitary matrices. The exponential map provides a correspondence between these spaces, and allows us to define a unitary matrix using n2 real coefficients relative to a basis of the Lie algebra. The parametrization is closed under additive updates of these coefficients, and thus provides a simple space in which to do gradient descent. We demonstrate the effectiveness of this parametrization on the problem of learning arbitrary unitary operators, comparing to several baselines and outperforming a recently-proposed lower-dimensional parametrization. We additionally use our parametrization to generalize a recently-proposed unitary recurrent neural network to arbitrary unitary matrices, using it to solve standard long-memory tasks.

Parameterized quantum circuits as machine learning models

“…For example, recurrent neural networks may suffer from the exploding gradient problem. This can be prevented by constraining the operations to be unitary and much work has been done to efficiently parameterize the unitary group [114,115]. PQC models have the advantage of naturally implementing unitary operations on an exponentially large vector space.…”

Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

“…Various algorithms have then been developed to train orthogonal RNNs. Typically, these works propose parametrizations of the orthogonal/unitary group that lead to computationally cheap operations, see (Arjovsky, Shah, and Bengio 2016;Wisdom et al 2016;Jing et al 2017;Hyland and Rätsch 2017;Mhammedi The Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) Helfrich, Willmott, and Ye 2018;Lezcano-Casado and Martínez-Rubio 2019;Maduranga, Helfrich, and Ye 2019). However, most of these works are focused on algorithmic contributions and do not contain convergence analyses.…”

Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs

“…During the development of neural networks, orthogonality was first shown to be useful in mitigating the vanishing or exploding gradients problem (Bengio, Simard, and Frasconi 1994), especially on recurrent neural networks (RNNs) (Pascanu, Mikolov, and Bengio 2013;Le, Jaitly, and Hinton 2015;Wisdom et al 2016;Arjovsky, Shah, and Bengio 2016;Jing et al 2017;Hyland and Rätsch 2017;Vorontsov et al 2017;Helfrich and Ye 2020). To improve the efficiency of the optimization algorithms with orthogonality, many techniques have been utilized, e.g., householder reflections (Mhammedi et al 2017), Cayley transform (Helfrich, Willmott, andYe 2018;Maduranga, Helfrich, and Ye 2019), and exponential-map-based parameterization (Lezcano Casado 2019;Lezcano-Casado and Martınez-Rubio 2019).…”