Maximizing the ratio of information to its cost in information theoretic competitive learning

Authors:
Ryotaro Kamimura;Sachiko Aida-Hyugaji
Affiliations:
Information Science Laboratory, Tokai University, Kanagawa, Japan;Information Science Laboratory, Tokai University, Kanagawa, Japan
Venue:
ICANN'05 Proceedings of the 15th international conference on Artificial neural networks: formal models and their applications - Volume Part II
Year:
2005

Citing 8
Cited 0

Self-Organization in a Perceptual Network

Computer
Competitive learning algorithms for vector quantization

Neural Networks
Feature discovery by competitive learning

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
On learning the past tenses of english verbs

Parallel distributed processing
The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals

Neural Computation
Faithful Representations and Topographic Maps: From Distortion- to Information-Based Self-Organization

Faithful Representations and Topographic Maps: From Distortion- to Information-Based Self-Organization
Local synaptic learning rules suffice to maximize mutual information in a linear network

Neural Computation
How to generate ordered maps by maximizing the mutual information between input and output signals

Neural Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we introduce costs in the framework of information maximization and try to maximize the ratio of information to its associated cost. We have shown that competitive learning is realized by maximizing mutual information between input patterns and competitive units. One shortcoming of the method is that maximizing information does not necessarily produce representations faithful to input patterns. Information maximizing primarily focuses on some parts of input patterns used to distinguish between patterns. Thus, we introduce the ratio of information to its cost that represents distance between input patterns and connection weights. By minimizing the ratio, final connection weights reflect well input patterns. We applied unsupervised information maximization to a voting attitude problem and supervised learning to a chemical data analysis. Experimental results confirmed that by minimizing the ratio, the cost is decreased with better generalization performance.