Relative Loss Bounds for Multidimensional Regression Problems
Machine Learning
General Convergence Results for Linear Discriminant Updates
Machine Learning
Learning Additive Models Online with Fast Evaluating Kernels
COLT '01/EuroCOLT '01 Proceedings of the 14th Annual Conference on Computational Learning Theory and and 5th European Conference on Computational Learning Theory
An introduction to boosting and leveraging
Advanced lectures on machine learning
Online learning of linear classifiers
Advanced lectures on machine learning
Tracking the best linear predictor
The Journal of Machine Learning Research
The Robustness of the p-Norm Algorithms
Machine Learning
Recursive Aggregation of Estimators by the Mirror Descent Algorithm with Averaging
Problems of Information Transmission
Online Passive-Aggressive Algorithms
The Journal of Machine Learning Research
A primal-dual perspective of online learning algorithms
Machine Learning
Leading strategies in competitive on-line prediction
Theoretical Computer Science
Surrogate regret bounds for proper losses
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Adaptive fuzzy filtering in a deterministic setting
IEEE Transactions on Fuzzy Systems
Learning Permutations with Exponential Weights
The Journal of Machine Learning Research
The Journal of Machine Learning Research
Leading strategies in competitive on-line prediction
ALT'06 Proceedings of the 17th international conference on Algorithmic Learning Theory
Online tracking of linear subspaces
COLT'06 Proceedings of the 19th annual conference on Learning Theory
COLT'06 Proceedings of the 19th annual conference on Learning Theory
COLT'05 Proceedings of the 18th annual conference on Learning Theory
| Hi-index | 0.00 |
We analyze and compare the well-known gradient descent algorithm and the more recent exponentiated gradient algorithm for training a single neuron with an arbitrary transfer function. Both algorithms are easily generalized to larger neural networks, and the generalization of gradient descent is the standard backpropagation algorithm. We prove worst-case loss bounds for both algorithms in the single neuron case. Since local minima make it difficult to prove worst case bounds for gradient-based algorithms, we must use a loss function that prevents the formation of spurious local minima. We define such a matching loss function for any strictly increasing differentiable transfer function and prove worst-case loss bounds for any such transfer function and its corresponding matching loss. The different forms of the two algorithms' bounds indicates that exponentiated gradient outperforms gradient descent when the inputs contain a large number of irrelevant components. Simulations on synthetic data confirm these analytical results