Boundedness and convergence of online gradient method with penalty for feedforward neural networks

Authors:
Huisheng Zhang;Wei Wu;Fei Liu;Mingchen Yao
Affiliations:
Applied Mathematics Department, Dalian University of Technology, Dalian, China and Department of Mathematics, Dalian Maritime University, Dalian, China;Applied Mathematics Department, Dalian University of Technology, Dalian, China;Department of Statistics, University of Missouri Columbia, Columbia, MO;Applied Mathematics Department, Dalian University of Technology, Dalian, China
Venue:
IEEE Transactions on Neural Networks
Year:
2009

Citing 9
Cited 10

Comparing biases for minimal network construction with back-propagation

Advances in neural information processing systems 1
Simplifying neural networks by soft weight-sharing

Neural Computation
Parameter convergence and learning curves for neural networks

Neural Computation
Neuro-Dynamic Programming

Neuro-Dynamic Programming
Gradient Convergence in Gradient methods with Errors

SIAM Journal on Optimization
Second-Order Learning Algorithm with Squared Penalty Term

Neural Computation
Convergent on-line algorithms for supervised learning in neural networks

IEEE Transactions on Neural Networks
Magnified gradient function with deterministic weight modification in adaptive learning

IEEE Transactions on Neural Networks
Deterministic convergence of an online gradient method for BP neural networks

IEEE Transactions on Neural Networks

A modified gradient-based neuro-fuzzy learning algorithm and its convergence

Information Sciences: an International Journal
Convergence analysis of three classes of split-complex gradient algorithms for complex-valued recurrent neural networks

Neural Computation
Convergence analysis of online gradient method for BP neural networks

Neural Networks
Deterministic convergence of conjugate gradient method for feedforward neural networks

Neurocomputing
Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks

Neurocomputing
Computational properties and convergence analysis of BPNN for cyclic and almost cyclic learning with penalty

Neural Networks
Letters: Convergence of gradient method with penalty for Ridge Polynomial neural network

Neurocomputing
A structure optimisation algorithm for feedforward neural network construction

Neurocomputing
Convergence of chaos injection-based batch backpropagation algorithm for feedforward neural networks

ISNN'13 Proceedings of the 10th international conference on Advances in Neural Networks - Volume Part I
Batch gradient method with smoothing L1/ 2 regularization for training of feedforward neural networks

Neural Networks

Quantified Score

Hi-index	0.01

Visualization

Abstract

In this brief, we consider an online gradient method with penalty for training feedforward neural networks. Specifically, the penalty is a term proportional to the norm of the weights. Its roles in the method are to control the magnitude of the weights and to improve the generalization performance of the network. By proving that the weights are automatically bounded in the network training with penalty, we simplify the conditions that are required for convergence of online gradient method in literature. A numerical example is given to support the theoretical analysis.