Divergence estimation for multidimensional densities via k-nearest-neighbor distances

Authors:
Qing Wang;Sanjeev R. Kulkarni;Sergio Verdú
Affiliations:
Credit Suisse Group, New York, NY;Department of Electrical Engineering, Princeton University, Princeton, NJ;Department of Electrical Engineering, Princeton University, Princeton, NJ
Venue:
IEEE Transactions on Information Theory
Year:
2009

Citing 13
Cited 10

A course in density estimation

A course in density estimation
Multidimensional binary search trees used for associative searching

Communications of the ACM
Texture Similarity Measure Using Kullback-Leibler Divergence between Gamma Distributions

ECCV '02 Proceedings of the 7th European Conference on Computer Vision-Part III
What Is the Nearest Neighbor in High Dimensional Spaces?

VLDB '00 Proceedings of the 26th International Conference on Very Large Data Bases
A divisive information theoretic feature clustering algorithm for text classification

The Journal of Machine Learning Research
An Efficient Image Similarity Measure Based on Approximations of KL-Divergence Between Two Gaussian Mixtures

ICCV '03 Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2
Gene Mapping and Marker Clustering Using Shannon's Mutual Information

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
On stationarity in Internet measurements through an information-theoretic lens

ICDEW '05 Proceedings of the 21st International Conference on Data Engineering Workshops
A Branch and Bound Algorithm for Computing k-Nearest Neighbors

IEEE Transactions on Computers
Kullback-leibler boosting

CVPR'03 Proceedings of the 2003 IEEE computer society conference on Computer vision and pattern recognition
Channel Identification: Secret Sharing Using Reciprocity in Ultrawideband Channels

IEEE Transactions on Information Forensics and Security - Part 1
Data-dependent kn-NN and kernel estimators consistent for arbitrary processes

IEEE Transactions on Information Theory
Divergence Estimation of Continuous Distributions Based on Data-Dependent Partitions

IEEE Transactions on Information Theory

Universal Estimation of Information Measures for Analog Sources

Foundations and Trends in Communications and Information Theory
On directed information theory and Granger causality graphs

Journal of Computational Neuroscience
A computationally efficient information estimator for weighted data

ICANN'11 Proceedings of the 21st international conference on Artificial neural networks - Volume Part II
Dimensionality reduction based on non-parametric mutual information

Neurocomputing
An unsupervised data projection that preserves the cluster structure

Pattern Recognition Letters
K Nearest Neighbor Equality: Giving equal chance to all existing classes

Information Sciences: an International Journal
Review: Divergence measures for statistical data processing-An annotated bibliography

Signal Processing
Information-theoretic measures of influence based on content dynamics

Proceedings of the sixth ACM international conference on Web search and data mining
Information estimators for weighted observations

Neural Networks
Pair-copula based mixture models and their application in clustering

Pattern Recognition

Quantified Score

Hi-index	754.84

Visualization

Abstract

A new universal estimator of divergence is presented for multidimensional continuous densities based on k-nearest-neighbor (k-NN) distances. Assuming independent and identically distributed (i.i.d.) samples, the new estimator is proved to be asymptotically unbiased and mean-square consistent. In experiments with high-dimensional data, the k-NN approach generally exhibits faster convergence than previous algorithms. It is also shown that the speed of convergence of the k-NN method can be further improved by an adaptive choice of k.