Model selection and weight sharing of multi-layer perceptrons

Authors:
Yusuke Tanahashi;Kazumi Saito;Ryohei Nakano
Affiliations:
Nagoya Institute of Technology, Nagoya, Japan;NTT Communication Science Laboratories, NTT Corporation, Soraku, Kyoto, Japan;Nagoya Institute of Technology, Nagoya, Japan
Venue:
KES'05 Proceedings of the 9th international conference on Knowledge-Based Intelligent Information and Engineering Systems - Volume Part IV
Year:
2005

Citing 2
Cited 2

Neural Networks for Pattern Recognition

Neural Networks for Pattern Recognition
Structuring Neural Networks through Bidirectional Clustering of Weights

DS '02 Proceedings of the 5th International Conference on Discovery Science

Bidirectional Clustering of MLP Weights for Finding Nominally Conditioned Polynomials

ICANN '09 Proceedings of the 19th International Conference on Artificial Neural Networks: Part II
Finding nominally conditioned multivariate polynomials using a four-layer perceptron having shared weights

KES'06 Proceedings of the 10th international conference on Knowledge-Based Intelligent Information and Engineering Systems - Volume Part II

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a method to learn and select a succinct multi-layer perceptron having shared weights. Weight sharing means a weight is allowed to have one of common weights. A near-zero common weight can be eliminated, called weight pruning. Our method iteratively merges and splits common weights based on 2nd-order criteria, escaping local optima through bidirectional clustering. Moreover, our method selects the optimal number of hidden units based on cross-validation. Our experiments showed that the proposed method can perfectly restore the original sharing structure for an artificial data set, and finds a small number of common weights for a real data set.