Unsupervised learning of hierarchical representations with convolutional deep belief networks

Authors:
Honglak Lee;Roger Grosse;Rajesh Ranganath;Andrew Y. Ng
Affiliations:
University of Michigan, Ann Arbor, MI;CSAIL, Massachusetts Institute of Technology, Cambridge, MA;Stanford University, Stanford, CA;Stanford University, Stanford, CA
Venue:
Communications of the ACM
Year:
2011

Citing 16
Cited 2

Training products of experts by minimizing contrastive divergence

Neural Computation
Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

CVPRW '04 Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12 - Volume 12
Shape Matching and Object Recognition Using Low Distortion Correspondences

CVPR '05 Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01
Topographic Product Models Applied to Natural Scene Statistics

Neural Computation
Multiclass Object Recognition with Sparse, Localized Features

CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories

CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2
SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition

CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2
A fast learning algorithm for deep belief nets

Neural Computation
An empirical evaluation of deep architectures on problems with many factors of variation

Proceedings of the 24th international conference on Machine learning
Self-taught learning: transfer learning from unlabeled data

Proceedings of the 24th international conference on Machine learning
Restricted Boltzmann machines for collaborative filtering

Proceedings of the 24th international conference on Machine learning
Backpropagation applied to handwritten zip code recognition

Neural Computation
Training restricted Boltzmann machines using approximations to the likelihood gradient

Proceedings of the 25th international conference on Machine learning
Deep learning via semi-supervised embedding

Proceedings of the 25th international conference on Machine learning
Large-scale deep unsupervised learning using graphics processors

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning

Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning

A hierarchical neural network architecture for classification

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part I
Hierarchical spatiotemporal feature extraction using recurrent online clustering

Pattern Recognition Letters

Quantified Score

Hi-index	48.22

Visualization

Abstract

There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks (DBNs); however, scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model that scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique that shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.