ReservoirPy: An Efficient and User-Friendly Library to Design Echo State Networks

Trouvain, Nathan; Pedrelli, Luca; Dinh, Thanh Trung; Hinaut, Xavier

doi:10.1007/978-3-030-61616-8_40

Cited by 40 publications

(21 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, twice more neurons and a bigger training corpus are needed to obtain significant performance. We verified this increased task difficulty with extensive hyperparameter search using hyperopt [17] and the graphical tool provided in ReservoirPy [16]. The fact that the representation proposed in Juven's model is dependent on the sentence structure (i.e.…”

Section: Experiments Iiib: Number Of Reservoir Hidden Unitsmentioning

confidence: 75%

“…To implement reservoir models in our study, we used ReservoirPy v0.2 [16]. It is an efficient library to design ESNs, which already supports offline and online training, as well as computational parallelization, fast reservoir initialization and other necessary utilities, such as optimized parameters search with hyperopt [17].…”

Section: A Reservoir Computing and Force Learningmentioning

confidence: 99%

See 1 more Smart Citation

Language Acquisition with Echo State Networks: Towards Unsupervised Learning

Dinh

Hinaut

2020

2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)

Self Cite

View full text Add to dashboard Cite

The modeling of children language acquisition with robots is a long quest paved with pitfalls. Recently a sentence parsing model learning in cross-situational conditions has been proposed: it learns from the robot visual representations. The model, based on random recurrent neural networks (i.e. reservoirs), can achieve significant performance after few hundreds of training examples, more quickly that what a theoretical model could do. In this study, we investigate the developmental plausibility of such model: (i) if it can learn to generalize from single-object sentence to double-object sentence; (ii) if it can use more plausible representations: (ii.a) inputs as sequence of phonemes (instead of words) and (ii.b) outputs fully independent from sentence structure (in order to enable purely unsupervised cross-situational learning). Interestingly, tasks (i) and (ii.a) are solved in a straightforward fashion, whereas task (ii.b) suggest that that learning with tensor representations is a more difficult task Index Terms-

show abstract

Section: Experiments Iiib: Number Of Reservoir Hidden Unitsmentioning

confidence: 75%

Section: A Reservoir Computing and Force Learningmentioning

confidence: 99%

Language Acquisition with Echo State Networks: Towards Unsupervised Learning

Dinh

Hinaut

2020

2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)

Self Cite

View full text Add to dashboard Cite

show abstract

“…We used the ReservoirPy toolbox 3 , for the implementation of ESNs [40]. The hyper-parameters were optimized through random search by Juven & Hinaut in [28] for one epoch of learning.…”

Section: The Esn Implementationmentioning

confidence: 99%

“…the coefficient of the matrix W out , are trained. This image was adapted from [40] with the authorisation of the authors.…”

Section: Reservoir Computingmentioning

confidence: 99%

A journey in ESN and LSTM visualisations on a language task

Variengien,

Hinaut

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Echo States Networks (ESN) and Long-Short Term Memory networks (LSTM) are two popular architectures of Recurrent Neural Networks (RNN) to solve machine learning task involving sequential data. However, little have been done to compare their performances and their internal mechanisms on a common task. In this work, we trained ESNs and LSTMs on a Cross-Situationnal Learning (CSL) task. This task aims at modelling how infants learn language: they create associations between words and visual stimuli in order to extract meaning from words and sentences. The results are of three kinds: performance comparison, internal dynamics analyses and visualization of latent space. (1) We found that both models were able to successfully learn the task: the LSTM reached the lowest error for the basic corpus, but the ESN was quicker to train. Furthermore, the ESN was able to outperform LSTMs on datasets more challenging without any further tuning needed. (2) We also conducted an analysis of the internal units activations of LSTMs and ESNs. Despite the deep differences between both models (trained or fixed internal weights), we were able to uncover similar inner mechanisms: both put emphasis on the units encoding aspects of the sentence structure. (3) Moreover, we present Recurrent States Space Visualisations (RSSviz), a method to visualize the structure of latent state space of RNNs, based on dimension reduction (using UMAP). This technique enables us to observe a fractal embedding of sequences in the LSTM. RSSviz is also useful for the analysis of ESNs (i) to spot difficult examples and (ii) to generate animated plots showing the evolution of activations across learning stages. Finally, we explore qualitatively how the RSSviz could provide an intuitive visualisation to understand the influence of hyperparameters on the reservoir dynamics prior to ESN training.

show abstract

“…As shown in [13], such a model shows the ability of producing realistic sounds (canary syllables), and represents an alternative to previously proposed vocal tract models. On the other hand, the sensory response function is defined as a Recurrent Neural Network (RNN) classifier [14] implemented with the ReservoirPy library [15]. For this study, the classifier and the GAN are pre-trained.…”

Section: Introductionmentioning

confidence: 99%

Canary Vocal Sensorimotor Model with RNN Decoder and Low-dimensional GAN Generator

Pagliarini

Leblois

Hinaut

2021

2021 IEEE International Conference on Development and Learning (ICDL)

Self Cite

View full text Add to dashboard Cite

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

ReservoirPy: An Efficient and User-Friendly Library to Design Echo State Networks

Cited by 40 publications

References 12 publications

Language Acquisition with Echo State Networks: Towards Unsupervised Learning

Language Acquisition with Echo State Networks: Towards Unsupervised Learning

A journey in ESN and LSTM visualisations on a language task

Canary Vocal Sensorimotor Model with RNN Decoder and Low-dimensional GAN Generator

Contact Info

Product

Resources

About