Improving recurrent neural network performance using transfer entropy

Authors:
Oliver Obst;Joschka Boedecker;Minoru Asada
Affiliations:
CSIRO ICT Centre, Adaptive Systems, Epping, NSW, Australia and School of Information Technologies, The University of Sydney, NSW, Australia;Department of Adaptive Machine Systems, Osaka University, Osaka, Japan and JST ERATO Asada Synergistic Intelligence Project, Osaka, Japan;Department of Adaptive Machine Systems, Osaka University, Osaka, Japan and JST ERATO Asada Synergistic Intelligence Project, Osaka, Japan
Venue:
ICONIP'10 Proceedings of the 17th international conference on Neural information processing: models and applications - Volume Part II
Year:
2010

Citing 6
Cited 0

Real-time computing without stable states: a new framework for neural computation based on perturbations

Neural Computation
2007 Special Issue: Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning

Neural Networks
Detecting non-trivial computation in complex dynamics

ECAL'07 Proceedings of the 9th European conference on Advances in artificial life
A gradient rule for the plasticity of a neuron’s intrinsic excitability

ICANN'05 Proceedings of the 15th international conference on Artificial Neural Networks: biological Inspirations - Volume Part I
Critical echo state networks

ICANN'06 Proceedings of the 16th international conference on Artificial Neural Networks - Volume Part I
Survey: Reservoir computing approaches to recurrent neural network training

Computer Science Review

Quantified Score

Hi-index	0.00

Visualization

Abstract

Reservoir computing approaches have been successfully applied to a variety of tasks. An inherent problem of these approaches, is, however, their variation in performance due to fixed random initialisation of the reservoir. Self-organised approaches like intrinsic plasticity have been applied to improve reservoir quality, but do not take the task of the system into account. We present an approach to improve the hidden layer of recurrent neural networks, guided by the learning goal of the system. Our reservoir adaptation optimises the information transfer at each individual unit, dependent on properties of the information transfer between input and output of the system. Using synthetic data, we show that this reservoir adaptation improves the performance of offline echo state learning and Recursive Least Squares Online Learning.