Elements of information theory
Elements of information theory
Inducing Features of Random Fields
IEEE Transactions on Pattern Analysis and Machine Intelligence
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Approximating MAP using Local Search
UAI '01 Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence
Iterative Markov Chain Monte Carlo Computation of Reference Priors and Minimax Risk
UAI '01 Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence
Sufficient dimensionality reduction
The Journal of Machine Learning Research
Geometric programming duals of channel capacity and rate distortion
IEEE Transactions on Information Theory
Multivariate information bottleneck
Neural Computation
Hi-index | 0.00 |
Exponential models of distributions are widely used in machine learning for classification and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classification tasks, mutual information is a more suitable information theoretic measure to be optimized. We show how the principle of minimum mutual information generalizes that of maximum entropy, and provides a comprehensive framework for building discriminative classifiers. A game theoretic interpretation of our approach is then given, and several generalization bounds provided. We present iterative algorithms for solving the minimum information problem and its convex dual, and demonstrate their performance on various classification tasks. The results show that minimum information classifiers outperform the corresponding maximum entropy models.