Optimal matrix compression yields storage capacity 1 for binary Willshaw associative memory

Authors:
Andreas Knoblauch
Affiliations:
Abteilung Neuroinformatik, Fakultät für Informatik, Universität Ulm, Ulm, Germany
Venue:
ICANN/ICONIP'03 Proceedings of the 2003 joint international conference on Artificial neural networks and neural information processing
Year:
2003

Citing 6
Cited 6

Bidirectional associative memories

IEEE Transactions on Systems, Man and Cybernetics
Information storage and effective data retrieval in sparse matrices

Neural Networks
Elements of information theory

Elements of information theory
Iterative retrieval of sparsely coded associative memory patterns

Neural Networks
Improved bidirectional retrieval of sparse patterns stored by Hebbian learning

Neural Networks
Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning

Neural Computation

Neural associative memory for brain modeling and information retrieval

Information Processing Letters - Special issue on applications of spiking neural networks
Memory capacities for synaptic and structural plasticity

Neural Computation
Neural associative memory for brain modeling and information retrieval

Information Processing Letters - Special issue on applications of spiking neural networks
Statistical implications of clipped Hebbian learning of cell assemblies

Neurocomputing
Neural networks letter: Tree-like hierarchical associative memory structures

Neural Networks
Neural associative memory with optimal bayesian learning

Neural Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

The classical binary Willshaw model of associative memory has an asymptotic storage capacity of ln 2 ≅ 0.7 which exceeds the capacities of other (e.g., Hopfield-like) models by far. However, its practical use is severely limited, since the asymptotic capacity is reached only for very large numbers n of neurons and for sparse patterns where the number k of one-entries must match a certain optimal value kopt(n) (typically kopt = log n). In this work I demonstrate that optimal compression of the binary memory matrix by a Huffman or Golomb code can increase the asymptotic storage capacity to 1. Moreover, it turns out that this happens for a very broad range of kbeing either ultra-sparse (e.g., k constant) or moderately-sparse (e.g., k = √n). A storage capacity in the range of ln 2 is already achieved for practical numbers of neurons.