Natural gradient works efficiently in learning
Neural Computation
Neural Networks for Pattern Recognition
Neural Networks for Pattern Recognition
| Hi-index | 0.00 |
Natural gradient learning is known to resolve the plateau problem, which is the main cause of slow learning speed of neural networks. The adaptive natural gradient learning, which is an adaptive method of realizing the natural gradien tlearning for neural networks, has also been developed and its practical advantage has been confirmed In this paper, we consider the generalization property of the natural gradient method Theoretically the standard gradient method and the natural gradient method has the same minimum in the error surface, thus the generalization performance should also be the same. However, in the practical sense, it is feasible that the natural gradient method gives smaller training error when the standard method stops learning in a plateau. In this case, the solutions that are practically obtained are different from each other, and their generalization performances also come to be different. Since these situations are very often in practical problems, it is necessary to compare the generalization property of the natural gradient learning method with the standard method. In this paper, we show a case that the practical generalization performance of the natural gradient learning is poorer than the standard gradient method, and try to solve the problem by including a regularization term in the natural gradient learning.