Density-free convergence properties of various estimators of entropy
Computational Statistics & Data Analysis - Special issue on statistical data analysis based on the L:0I1:0E norm and relate
Independent component analysis, a new concept?
Signal Processing - Special issue on higher order statistics
Machine Learning
ACM Computing Surveys (CSUR)
An approximate method for generating asymmetric random variables
Communications of the ACM
Machine Learning
Estimation of entropy and mutual information
Neural Computation
ICA using spacings estimates of entropy
The Journal of Machine Learning Research
IEEE Transactions on Knowledge and Data Engineering
The cross entropy method for classification
ICML '05 Proceedings of the 22nd international conference on Machine learning
Nonparametric Quantile Estimation
The Journal of Machine Learning Research
Tailoring density estimation via reproducing kernel moment matching
Proceedings of the 25th international conference on Machine learning
Divergence estimation for multidimensional densities via k-nearest-neighbor distances
IEEE Transactions on Information Theory
A computationally efficient information estimator for weighted data
ICANN'11 Proceedings of the 21st international conference on Artificial neural networks - Volume Part II
Hi-index | 0.00 |
The Shannon information content is a valuable numerical characteristic of probability distributions. The problem of estimating the information content from an observed dataset is very important in the fields of statistics, information theory, and machine learning. The contribution of the present paper is in proposing information estimators, and showing some of their applications. When the given data are associated with weights, each datum contributes differently to the empirical average of statistics. The proposed estimators can deal with this kind of weighted data. Similar to other conventional methods, the proposed information estimator contains a parameter to be tuned, and is computationally expensive. To overcome these problems, the proposed estimator is further modified so that it is more computationally efficient and has no tuning parameter. The proposed methods are also extended so as to estimate the cross-entropy, entropy, and Kullback-Leibler divergence. Simple numerical experiments show that the information estimators work properly. Then, the estimators are applied to two specific problems, distribution-preserving data compression, and weight optimization for ensemble regression.