2007 Special Issue: Optimization and applications of echo state networks with leaky- integrator neurons

  • Authors:
  • Herbert Jaeger;Mantas Lukoševičius;Dan Popovici;Udo Siewert

  • Affiliations:
  • Jacobs University Bremen, School of Engineering and Science, 28759 Bremen, Germany;Jacobs University Bremen, School of Engineering and Science, 28759 Bremen, Germany;Jacobs University Bremen, School of Engineering and Science, 28759 Bremen, Germany;Planet intelligent systems GmbH, Residence Park 1-7, D-19065 Raben Steinfeld, Germany

  • Venue:
  • Neural Networks
  • Year:
  • 2007

Quantified Score

Hi-index 0.01

Visualization

Abstract

Standard echo state networks (ESNs) are built from simple additive units with a sigmoid activation function. Here we investigate ESNs whose reservoir units are leaky integrator units. Units of this type have individual state dynamics, which can be exploited in various ways to accommodate the network to the temporal characteristics of a learning task. We present stability conditions, introduce and investigate a stochastic gradient descent method for the optimization of the global learning parameters (input and output feedback scalings, leaking rate, spectral radius) and demonstrate the usefulness of leaky-integrator ESNs for (i) learning very slow dynamic systems and replaying the learnt system at different speeds, (ii) classifying relatively slow and noisy time series (the Japanese Vowel dataset - here we obtain a zero test error rate), and (iii) recognizing strongly time-warped dynamic patterns.