Training products of experts by minimizing contrastive divergence
Neural Computation
Energy-based models for sparse overcomplete representations
The Journal of Machine Learning Research
A fast learning algorithm for deep belief nets
Neural Computation
Self-taught learning: transfer learning from unlabeled data
Proceedings of the 24th international conference on Machine learning
Classification using discriminative restricted Boltzmann machines
Proceedings of the 25th international conference on Machine learning
| Hi-index | 0.00 |
It has been shown that the Deep Belief Network is good at modeling input distribution, and can be trained efficiently by the greedy layer-wise unsupervised learning. Hoglak Lee et al. (2008) introduced a sparse variant of the Deep Belief Network, which applied the Gaussian linear units to model the input data with a sparsity constraint. However, it takes much more weight updates to train the RBM (Restricted Boltzmann Machine) with Gaussian visible units, and the reconstruction error is much larger than training an RBM with binary visible units. Here, we propose another version of Sparse Deep Belief Net which applies the differentiable sparse coding method to train the first level of the deep network, and then train the higher layers with RBM. This hybrid model, combining the advantage of the Deep architecture and the sparse coding model, leads to state-of-the-art performance on the classification of handwritten digits.