Connectionist learning procedures
Artificial Intelligence
Introduction to artificial neural systems
Introduction to artificial neural systems
Neural networks and the bias/variance dilemma
Neural Computation
Structural learning with forgetting
Neural Networks
A penalty-function approach for pruning feedforward neural networks
Neural Computation
On-line learning in neural networks
On-line learning in neural networks
Atomic Decomposition by Basis Pursuit
SIAM Journal on Scientific Computing
Neural Networks: A Comprehensive Foundation
Neural Networks: A Comprehensive Foundation
Deterministic convergence of an online gradient method for neural networks
Journal of Computational and Applied Mathematics - Selected papers of the international symposium on applied mathematics, August 2000, Dalian, China
Second-Order Learning Algorithm with Squared Penalty Term
Neural Computation
When does online BP training converge?
IEEE Transactions on Neural Networks
Uncertainty principles and ideal atomic decomposition
IEEE Transactions on Information Theory
A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning
IEEE Transactions on Neural Networks
On performance evaluation in online approximation for control
IEEE Transactions on Neural Networks
Deterministic convergence of an online gradient method for BP neural networks
IEEE Transactions on Neural Networks
IEEE Transactions on Neural Networks
| Hi-index | 0.01 |
Minimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L"2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L"1"/"2 regularization term. For normal L"1"/"2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L"1"/"2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L"1"/"2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L"2 and normal L"1"/"2 regularizations respectively.