Learning in linear neural networks: a survey
IEEE Transactions on Neural Networks
Average-Case Analysis of Classification Algorithms for Boolean Functions and Decision Trees
ALT '00 Proceedings of the 11th International Conference on Algorithmic Learning Theory
Localized bayes estimation for non-identifiable models
ICONIP'06 Proceedings of the 13 international conference on Neural Information Processing - Volume Part I
| Hi-index | 0.00 |
The statistical asymptotic theory is often used in theoretical results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator (MLE) as an normal distribution. However, in layered models such as neural networks, the regularity condition of the asymptotic theory is not necessarily satisfied. The true parameter is not identifiable, if the target function can be realized by a network of smaller size than the size of the model. There has been little known on the behavior of the MLE in these cases of neural networks. In this paper, we analyze the expectation of the generalization error of three-layer linear neural networks, and elucidate a strange behavior in unidentifiable cases. We show that the expectation of the generalization error in the unidentifiable cases is larger than what is given by the usual asymptotic theory, and dependent on the rank of the target function.