Competitive learning by information maximization: eliminating dead neurons in competitive learning

Authors:
Ryotaro Kamimura
Affiliations:
Information Science Laboratory and Future Science and Technology Joint Research Center, Tokai University, Hiratsuka, Kanagawa, Japan
Venue:
ICANN/ICONIP'03 Proceedings of the 2003 joint international conference on Artificial neural networks and neural information processing
Year:
2003

Citing 5
Cited 0

Self-Organization in a Perceptual Network

Computer
Competitive learning algorithms for vector quantization

Neural Networks
Unsupervised mutual information criterion for elimination of overtraining in supervised multilayer networks

Neural Computation
Faithful Representations and Topographic Maps: From Distortion- to Information-Based Self-Organization

Faithful Representations and Topographic Maps: From Distortion- to Information-Based Self-Organization
How to generate ordered maps by maximizing the mutual information between input and output signals

Neural Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we propose a new information theoretic competitive learning method. In realizing competition, neither the winner-take-all algorithm nor the lateral inhibition is used. Instead, the new method is based upon mutual information maximization between input patterns and competitive units. In maximizing mutual information, the entropy of competitive units is increased as much as possible. This means that all competitive units must equally be used in our framework. Thus, no under-utilized neurons (dead neurons) are generated. We applied our method to a simple artificial data problem and an actual road classification problem. In both cases, experimental results confirmed that the new method can produce the final solutions almost independently of initial conditions, and classification performance is significantly improved.