Higher order contractive auto-encoder

Authors:
Salah Rifai;Grégoire Mesnil;Pascal Vincent;Xavier Muller;Yoshua Bengio;Yann Dauphin;Xavier Glorot
Affiliations:
Dept. IRO, Université de Montréal. Montréal, QC, Canada;Dept. IRO, Université de Montréal. Montréal, QC, Canada and LITIS EA 4108, France;Dept. IRO, Université de Montréal. Montréal, QC, Canada;Dept. IRO, Université de Montréal. Montréal, QC, Canada;Dept. IRO, Université de Montréal. Montréal, QC, Canada;Dept. IRO, Université de Montréal. Montréal, QC, Canada;Dept. IRO, Université de Montréal. Montréal, QC, Canada
Venue:
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part II
Year:
2011

Citing 7
Cited 2

A fast learning algorithm for deep belief nets

Neural Computation
An empirical evaluation of deep architectures on problems with many factors of variation

Proceedings of the 24th international conference on Machine learning
Extracting and composing robust features with denoising autoencoders

Proceedings of the 25th international conference on Machine learning
Deep learning via semi-supervised embedding

Proceedings of the 25th international conference on Machine learning
Learning Deep Architectures for AI

Foundations and Trends® in Machine Learning
Semi-Supervised Learning

Semi-Supervised Learning
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

The Journal of Machine Learning Research

Nobody likes Mondays: foreground detection and behavioral patterns analysis in complex urban scenes

Proceedings of the 4th ACM/IEEE international workshop on Analysis and retrieval of tracked events and motion in imagery stream
Demystifying sparse rectified auto-encoders

Proceedings of the Fourth Symposium on Information and Communication Technology

Quantified Score

Hi-index	0.00

Visualization

Abstract

We propose a novel regularizer when training an autoencoder for unsupervised feature extraction. We explicitly encourage the latent representation to contract the input space by regularizing the norm of the Jacobian (analytically) and the Hessian (stochastically) of the encoder's output with respect to its input, at the training points. While the penalty on the Jacobian's norm ensures robustness to tiny corruption of samples in the input space, constraining the norm of the Hessian extends this robustness when moving further away from the sample. From a manifold learning perspective, balancing this regularization with the auto-encoder's reconstruction objective yields a representation that varies most when moving along the data manifold in input space, and is most insensitive in directions orthogonal to the manifold. The second order regularization, using the Hessian, penalizes curvature, and thus favors smooth manifold. We show that our proposed technique, while remaining computationally efficient, yields representations that are significantly better suited for initializing deep architectures than previously proposed approaches, beating state-of-the-art performance on a number of datasets.