Reservoir Computing Universality With Stochastic Inputs

Abstract

The universal approximation properties with respect to L p -type criteria of three important families of reservoir computers with stochastic discrete-time semi-infinite inputs are shown. First, it is proved that linear reservoir systems with either polynomial or neural network readout maps are universal. More importantly, it is proved that the same property holds for two families with linear readouts, namely, trigonometric state-affine systems and echo state networks, which are the most widely used reservoir systems in applications. The linearity in the readouts is a key feature in supervised machine learning applications. It guarantees that these systems can be used in high-dimensional situations and in the presence of large datasets. The L p criteria used in this paper allow the formulation of universality results that do not necessarily impose almost sure uniform boundedness in the inputs or the fading memory property in the filter that needs to be approximated.

Discrete-Time Signatures and Randomness in Reservoir Computing

“…In this article, we provide additional insight on the randomization question for another family of RC systems, namely, for the nonhomogeneous state-affine systems (SAS). These systems have been introduced and proved to be universal approximants in [36] and [37]. We here show that they also have this universality property when they are randomly generated.…”

“…It has been shown in [36] and [37] that SAS systems are universal approximants in the fading memory and in the L p -integrable categories in the sense that, given a filter in any of those two categories, there exists an SAS system of type ( 6) that uniformly, or in the L p -sense, approximates it.…”

Recent advances in physical reservoir computing: A review

“…It was shown that any given time-invariant filter (a transformation from u(·) to y(·)) with the fading memory property can be approximated by LSMs to any degree of precision, if L M is chosen from a class of time-invariant filters with fading memory that has a point-wise separation property and f M is chosen from a class of functions that satisfies an approximation property (Maass et al, 2002;. Recent studies have treated the universal approximation property of LSMs and other RC systems in a more mathematically rigorous way (Grigoryeva & Ortega, 2018;Gonon & Ortega, 2018).…”

Reservoir-Computing Based Equalization With Optical Pre-Processing for Short-Reach Optical Transmission

“…Concerning f act , the properties of ESN, including its universal approximation property, are strictly defined for sigmoid functions [26], [34], [35], [36]. However, in a number of demonstrations [13], [33], using alternative functions, even linear functions, to replace a sigmoid has been shown to be computationally effective.…”