Systemical convergence rate analysis of convex incremental feedforward neural networks

Authors:
Lei Chen;Guang-Bin Huang;Hung Keng Pung
Affiliations:
Network Systems and Service Lab, Department of Computer Science, National University of Singapore, Singapore;Network Systems and Service Lab, Department of Computer Science, National University of Singapore, Singapore;Network Systems and Service Lab, Department of Computer Science, National University of Singapore, Singapore
Venue:
Neurocomputing
Year:
2009

Citing 13
Cited 5

Multilayer feedforward networks are universal approximators

Neural Networks
Approximation capabilities of multilayer feedforward networks

Neural Networks
A resource-allocating network for function interpolation

Neural Computation
Approximation of functions on a compact set by finite sums of a sigmoid function without scaling

Neural Networks
Original Contribution: Approximation of continuous functions on Rd by linear combinations of shifted rotations of a sigmoid function with and without scaling

Neural Networks
Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Neural Networks
A sequential learning scheme for function approximation using minimal radial basis function neural networks

Neural Computation
Effiicient BackProp

Neural Networks: Tricks of the Trade, this book is an outgrowth of a 1996 NIPS workshop
Letters: Convex incremental extreme learning machine

Neurocomputing
Efficient agnostic learning of neural networks with bounded fan-in

IEEE Transactions on Information Theory - Part 2
Universal approximation bounds for superpositions of a sigmoidal function

IEEE Transactions on Information Theory
Objective functions for training new hidden units in constructive neural networks

IEEE Transactions on Neural Networks
Universal approximation using incremental constructive feedforward networks with random hidden nodes

IEEE Transactions on Neural Networks

Extreme learning machine based genetic algorithm and its application in power system economic dispatch

Neurocomputing
Evolutionary ELM wrapper feature selection for Alzheimer's disease CAD on anatomical brain MRI

Neurocomputing
Extreme learning machines for soybean classification in remote sensing hyperspectral images

Neurocomputing
Hybrid extreme rotation forest

Neural Networks
Applications of Hybrid Extreme Rotation Forests for image segmentation

International Journal of Hybrid Intelligent Systems

Quantified Score

Hi-index	0.01

Visualization

Abstract

In this paper, we systemically investigate several convex incremental feedforward neural networks. Firstly, we prove the universal approximation and the convergence rate of a generalized convex incremental (GCI) structure, which provides us a wider parameter selection. Second, according to the convergence rate proof of GCI, we further prove the convergence rate of a best convex incremental (BCI) structure, moreover its proof also illustrates that BCI can achieve a better generalization performance than GCI. But we should note that the hidden neurons of BCI and GCI both are constructed on the maximum principle (not random). Next, we introduce the random neuron conception based on CI-ELM (convex incremental extreme learning machines), and further propose an alternative algorithm (improved CI-ELM, ICI-ELM) between CI-ELM and BCI, which removes the ''useless'' neurons in CI-ELM and improves the efficiency of neural networks. ICI-ELM randomly generates a group of parameters, among which we determine the best parameters leading to the smallest residual error. Therefore ICI-ELM can achieve a faster convergence rate than CI-ELM, meanwhile it still retains the same convergence rate as BCI. On the other hand, ICI-ELM also provides an alternative scheme to replace conventional gradient methods, which are only suitable for differential functions and often achieves local minima. The experimental results based on several benchmark regression problems support our claims.