Analysis of Tikhonov regularization for function approximation by neural networks

Authors:
Martin Burger;Andreas Neubauer
Affiliations:
Institut für Industriemathematik, Johannes Kepler Universität, A-4040 Linz, Austria;Institut für Industriemathematik, Johannes Kepler Universität, A-4040 Linz, Austria
Venue:
Neural Networks
Year:
2003

Citing 10
Cited 6

Multilayer feedforward networks are universal approximators

Neural Networks
Approximation by ridge functions and neural networks with one hidden layer

Journal of Approximation Theory
Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Neural Networks
Regular Article: Degree of Approximation by Neural and Translation Networks with a Single Hidden Layer

Advances in Applied Mathematics
Regularization theory and neural networks architectures

Neural Computation
Training with noise is equivalent to Tikhonov regularization

Neural Computation
Uniform approximation by neural networks

Journal of Approximation Theory
Error bounds for approximation with neural networks

Journal of Approximation Theory
Neural Networks for Pattern Recognition

Neural Networks for Pattern Recognition
Convergence Rates of Approximation by Translates

Convergence Rates of Approximation by Translates

Regularized greedy algorithms for network training with data noise

Computing
Nondeterministic Discretization of Weights Improves Accuracy of Neural Networks

ECML '07 Proceedings of the 18th European conference on Machine Learning
Dealing with limited data in ballistic impact scenarios: an empirical comparison of different neural network approaches

Applied Intelligence
An optimization methodology for machine learning strategies and regression problems in ballistic impact scenarios

Applied Intelligence
Quantile based decision making rule of the neural networks committee for ill-posed approximation problems

Neurocomputing
Neural network modeling of vector multivariable functions in ill-posed approximation problems

Journal of Computer and Systems Sciences International

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper is devoted to the convergence and stability analysis of Tikhonov regularization for function approximation by a class of feed-forward neural networks with one hidden layer and linear output layer. We investigate two frequently used approaches, namely regularization by output smoothing and regularization by weight decay, as well as a combination of both methods to combine their advantages. We show that in all cases stable approximations are obtained converging to the approximated function in a desired Sobolev space as the noise in the data tends to zero (in the weaker L2-norm) if the regularization parameter and the number of units in the network are chosen appropriately. Under additional smoothness assumptions we are able to show convergence rates results in terms of the noise level and the number of units in the network.In addition, we show how the theoretical results can be applied to the important classes of perceptrons with one hidden layer and to translation networks. Finally, the performance of the different approaches is compared in some numerical examples.