Bounding the bias of contrastive divergence learning

Authors:
Asia Fischer;Christian Igel
Affiliations:
Institut für Neuroinformatik, Ruhr-Universität Bochum, Bochum, Germany;Department of Computer Science, University of Copenhagen, Copenhagen Ø, Denmark
Venue:
Neural Computation
Year:
2011

Citing 5
Cited 2

Information processing in dynamical systems: foundations of harmony theory

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
Training products of experts by minimizing contrastive divergence

Neural Computation
A fast learning algorithm for deep belief nets

Neural Computation
Justifying and generalizing contrastive divergence

Neural Computation
Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines

ICANN'10 Proceedings of the 20th international conference on Artificial neural networks: Part III

Training restricted Boltzmann machines: An introduction

Pattern Recognition
The flip-the-state transition operator for restricted Boltzmann machines

Machine Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.