Estimating divergence functionals and the likelihood ratio by convex risk minimization

Authors:
XuanLong Nguyen;Martin J. Wainwright;Michael I. Jordan
Affiliations:
Department of Statistics, University of Michigan, Ann Arbor, MI;Department of Electrical Engineering Computer Science and the Department of Statistics, University of California, Berkeley, CA;Department of Electrical Engineering Computer Science and the Department of Statistics, University of California, Berkeley, CA
Venue:
IEEE Transactions on Information Theory
Year:
2010

Citing 9
Cited 9

Density-free convergence properties of various estimators of entropy

Computational Statistics & Data Analysis - Special issue on statistical data analysis based on the L:0I1:0E norm and relate
Elements of information theory

Elements of information theory
Independent component analysis, a new concept?

Signal Processing - Special issue on higher order statistics
The covering number in learning theory

Journal of Complexity
Kernel Methods for Pattern Analysis

Kernel Methods for Pattern Analysis
Parametric estimation and tests through divergences and the duality technique

Journal of Multivariate Analysis
Some inequalities for information divergence and related measures of discrimination

IEEE Transactions on Information Theory
Divergence Estimation of Continuous Distributions Based on Data-Dependent Partitions

IEEE Transactions on Information Theory
On Divergences and Informations in Statistics and Information Theory

IEEE Transactions on Information Theory

Direct density-ratio estimation with dimensionality reduction via least-squares hetero-distributional subspace search

Neural Networks
Least-squares two-sample test

Neural Networks
Review: Divergence measures for statistical data processing-An annotated bibliography

Signal Processing
Computational complexity of kernel-based density-ratio estimation: a condition number analysis

Machine Learning
Sufficient dimension reduction via squared-loss mutual information estimation

Neural Computation
Change-point detection in time-series data by relative density-ratio estimation

Neural Networks
Density-difference estimation

Neural Computation
Change-point detection with feature selection in high-dimensional time-series data

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Semi-supervised learning of class balance under class-prior change by distribution matching

Neural Networks

Quantified Score

Hi-index	754.85

Visualization

Abstract

We develop and analyze M-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a nonasymptotic variational characterization of f-divergences, which allows the problem of estimating divergences to be tackled via convex empirical risk optimization. The resulting estimators are simple to implement, requiring only the solution of standard convex programs. We present an analysis of consistency and convergence for these estimators. Given conditions only on the ratios of densities, we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes. We derive an efficient optimization algorithm for computing our estimates, and illustrate their convergence behavior and practical viability by simulations.