How dependencies between successive examples affect on-line learning

Authors:
Wim Wiegerinck;Tom Heskes
Affiliations:
RWCP Novel Functions SNN Lab., Dept. of Medical Physics and Biophysics, Univ. of Nijmegen, Geert G ...;RWCP Novel Functions SNN Lab., Dept. of Medical Physics and Biophysics, Univ. of Nijmegen, Geert G ...
Venue:
Neural Computation
Year:
1996

Citing 12
Cited 0

Convergence properties of Kohonen's topology conserving maps: fluctuations, stability, and dimension selection

Biological Cybernetics
Adaptive algorithms and stochastic approximations

Adaptive algorithms and stochastic approximations
Introduction to the theory of neural computation

Introduction to the theory of neural computation
Original Contribution: Convergence and divergence in neural networks: Processing of chaos and biological analogy

Neural Networks
Pedagogical pattern selection strategies

Neural Networks
Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima

Neural Computation
Neural Networks: A Comprehensive Foundation

Neural Networks: A Comprehensive Foundation
Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times

Advances in Neural Information Processing Systems 5, [NIPS Conference]
Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria

Advances in Neural Information Processing Systems 5, [NIPS Conference]
Increased Complexity Training

IWANN '93 Proceedings of the International Workshop on Artificial Neural Networks: New Trends in Neural Computation
Generalization Performance of Overtrained Back-Propagation Networks

Proceedings of the EURASIP Workshop 1990 on Neural Networks
A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning

IEEE Transactions on Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

We study the dynamics of on-line learning for a large class of neural networks and learning rules, including backpropagation for multilayer perceptrons. In this paper, we focus on the case where successive examples are dependent, and we analyze how these dependencies affect the learning process. We define the representation error and the prediction error. The representation error measures how well the environment is represented by the network after learning. The prediction error is the average error that a continually learning network makes on the next example. In the neighborhood of a local minimum of the error surface, we calculate these errors. We find that the more predictable the example presentation, the higher the representation error, i.e., the less accurate the asymptotic representation of the whole environment. Furthermore we study the learning process in the presence of a plateau. Plateaus are flat spots on the error surface, which can severely slow down the learning process. In particular, they are notorious in applications with multilayer perceptrons. Our results, which are confirmed by simulations of a multilayer perceptron learning a chaotic time series using backpropagation, explain how dependencies between examples can help the learning process to escape from a plateau.