Nondeterministic Discretization of Weights Improves Accuracy of Neural Networks

Authors:
Marcin Wojnarski
Affiliations:
Warsaw University, Faculty of Mathematics, Informatics and Mechanics, ul. Banacha 2, 02-097 Warszawa, Poland
Venue:
ECML '07 Proceedings of the 18th European conference on Machine Learning
Year:
2007

Citing 9
Cited 0

Creating artificial neural networks that generalize

Neural Networks
Artificial intelligence: a modern approach

Artificial intelligence: a modern approach
Training with noise is equivalent to Tikhonov regularization

Neural Computation
Machine learning, neural and statistical classification

Machine learning, neural and statistical classification
Pattern Recognition and Neural Networks

Pattern Recognition and Neural Networks
Analysis of Tikhonov regularization for function approximation by neural networks

Neural Networks
LTF-C: architecture, training algorithm and applications of new neural classifier

Fundamenta Informaticae
Advances in Minimum Description Length: Theory and Applications (Neural Information Processing)

Advances in Minimum Description Length: Theory and Applications (Neural Information Processing)
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)

Quantified Score

Hi-index	0.00

Visualization

Abstract

The paper investigates modification of backpropagation algorithm, consisting of discretization of neural network weights after each training cycle. This modification, aimed at overfitting reduction, restricts the set of possible values of weights to a discrete subset of real numbers, leading to much better generalization abilities of the network. This, in turn, leads to higher accuracy and a decrease in error rate by over 50% in extreme cases (when overfitting is high).Discretization is performed nondeterministically, so as to keep expected value of discretized weight equal to original value. In this way, global behavior of original algorithm is preserved. The presented method of discretization is general and may be applied to other machine-learning algorithms. It is also an example of how an algorithm for continuous optimization can be successfully applied to optimization over discrete spaces. The method was evaluated experimentally in WEKA environment using two real-world data sets from UCI repository.