Matrix analysis
On the learnability of discrete distributions
STOC '94 Proceedings of the twenty-sixth annual ACM symposium on Theory of computing
Estimating a mixture of two product distributions
COLT '99 Proceedings of the twelfth annual conference on Computational learning theory
Latent semantic indexing: a probabilistic analysis
Journal of Computer and System Sciences - Special issue on the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on principles of database systems
Sampling algorithms: lower bounds and applications
STOC '01 Proceedings of the thirty-third annual ACM symposium on Theory of computing
STOC '01 Proceedings of the thirty-third annual ACM symposium on Theory of computing
Evolutionary Trees Can be Learned in Polynomial Time in the Two-State General Markov Model
SIAM Journal on Computing
Learning Mixtures of Gaussians
FOCS '99 Proceedings of the 40th Annual Symposium on Foundations of Computer Science
Spectral Partitioning of Random Graphs
FOCS '01 Proceedings of the 42nd IEEE symposium on Foundations of Computer Science
The Journal of Machine Learning Research
A spectral algorithm for learning mixture models
Journal of Computer and System Sciences - Special issue on FOCS 2002
Learning nonsingular phylogenies and hidden Markov models
Proceedings of the thirty-seventh annual ACM symposium on Theory of computing
Spectral norm of random matrices
Proceedings of the thirty-seventh annual ACM symposium on Theory of computing
On Learning Mixtures of Heavy-Tailed Distributions
FOCS '05 Proceedings of the 46th Annual IEEE Symposium on Foundations of Computer Science
A Probabilistic Analysis of EM for Mixtures of Separated, Spherical Gaussians
The Journal of Machine Learning Research
A rigorous analysis of population stratification with limited data
SODA '07 Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms
Using mixture models for collaborative filtering
Journal of Computer and System Sciences
Learning Mixtures of Product Distributions over Discrete Domains
SIAM Journal on Computing
The Spectral Method for General Mixture Models
SIAM Journal on Computing
Isotropic PCA and Affine-Invariant Clustering
FOCS '08 Proceedings of the 2008 49th Annual IEEE Symposium on Foundations of Computer Science
Concentration of Measure for the Analysis of Randomized Algorithms
Concentration of Measure for the Analysis of Randomized Algorithms
Latent class models for collaborative filtering
IJCAI'99 Proceedings of the 16th international joint conference on Artificial intelligence - Volume 2
Efficiently learning mixtures of two Gaussians
Proceedings of the forty-second ACM symposium on Theory of computing
Settling the Polynomial Learnability of Mixtures of Gaussians
FOCS '10 Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science
Polynomial Learning of Distribution Families
FOCS '10 Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science
Probabilistic latent semantic analysis
UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Learning k-modal distributions via testing
Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms
PAC learning axis-aligned mixtures of gaussians with no separation assumption
COLT'06 Proceedings of the 19th annual conference on Learning Theory
Communications of the ACM
On spectral learning of mixtures of distributions
COLT'05 Proceedings of the 18th annual conference on Learning Theory
The Inverse Moment Problem for Convex Polytopes
Discrete & Computational Geometry
Learning Topic Models -- Going beyond SVD
FOCS '12 Proceedings of the 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science
Hi-index | 0.00 |
We give an algorithm for learning a mixture of unstructured distributions. This problem arises in various unsupervised learning scenarios, for example in learning topic models from a corpus of documents spanning several topics. We show how to learn the constituents of a mixture of k arbitrary distributions over a large discrete domain [n]={1, 2, ...,n} and the mixture weights, using O(n polylog n) samples. (In the topic-model learning setting, the mixture constituents correspond to the topic distributions.) This task is information-theoretically impossible for k 1 under the usual sampling process from a mixture distribution. However, there are situations (such as the above-mentioned topic model case) in which each sample point consists of several observations from the same mixture constituent. This number of observations, which we call the "sampling aperture", is a crucial parameter of the problem. We obtain the first bounds for this mixture-learning problem without imposing any assumptions on the mixture constituents. We show that efficient learning is possible exactly at the information-theoretically least-possible aperture of 2k-1. Thus, we achieve near-optimal dependence on n and optimal aperture. While the sample-size required by our algorithm depends exponentially on k, we prove that such a dependence is unavoidable when one considers general mixtures. A sequence of tools contribute to the algorithm, such as concentration results for random matrices, dimension reduction, moment estimations, and sensitivity analysis.