Learning two-layer contractive encodings

Authors:
Hannes Schulz;Sven Behnke
Affiliations:
Institut für Informatik VI, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany;Institut für Informatik VI, Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany
Venue:
ICANN'12 Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part I
Year:
2012

Citing 6
Cited 0

A fast learning algorithm for deep belief nets

Neural Computation
Representational power of restricted boltzmann machines and deep belief networks

Neural Computation
Why Does Unsupervised Pre-training Help Deep Learning?

The Journal of Machine Learning Research
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

The Journal of Machine Learning Research
On the expressive power of deep architectures

ALT'11 Proceedings of the 22nd international conference on Algorithmic learning theory
Random search for hyper-parameter optimization

The Journal of Machine Learning Research

Quantified Score

Hi-index	0.00

Visualization

Abstract

Unsupervised learning of feature hierarchies is often a good initialization for supervised training of deep architectures. In existing deep learning methods, these feature hierarchies are built layer by layer in a greedy fashion using auto-encoders or restricted Boltzmann machines. Both yield encoders, which compute linear projections followed by a smooth thresholding function. In this work, we demonstrate that these encoders fail to find stable features when the required computation is in the exclusive-or class. To overcome this limitation, we propose a two-layer encoder which is not restricted in the type of features it can learn. The proposed encoder can be regularized by an extension of previous work on contractive regularization. We demonstrate the advantages of two-layer encoders qualitatively, as well as on commonly used benchmark datasets.