Fast Parallel Estimation of High Dimensional Information Theoretical Quantities with Low Dimensional Random Projection Ensembles

Authors:
Zoltán Szabó;András Lőrincz
Affiliations:
Department of Information Systems, Eötvös Loránd University, Budapest, Hungary H-1117;Department of Information Systems, Eötvös Loránd University, Budapest, Hungary H-1117
Venue:
ICA '09 Proceedings of the 8th International Conference on Independent Component Analysis and Signal Separation
Year:
2009

Citing 11
Cited 1

Blind separation of sources, Part 1: an adaptive algorithm based on neuromimetic architecture

Signal Processing
An optimal algorithm for approximate nearest neighbor searching fixed dimensions

Journal of the ACM (JACM)
ICA using spacings estimates of entropy

The Journal of Machine Learning Research
Uniqueness of complex and multidimensional independent component analysis

Signal Processing
Beyond independent components: trees and clusters

The Journal of Machine Learning Research
An algorithmic theory of learning: Robust concepts and random projection

Machine Learning
Undercomplete Blind Subspace Deconvolution

The Journal of Machine Learning Research
On variants of the Johnson–Lindenstrauss lemma

Random Structures & Algorithms
Independent process analysis without a priori dimensional information

ICA'07 Proceedings of the 7th international conference on Independent component analysis and signal separation
Independent subspace analysis using k-nearest neighborhood distances

ICANN'05 Proceedings of the 15th international conference on Artificial neural networks: formal models and their applications - Volume Part II
Cross-Entropy optimization for independent process analysis

ICA'06 Proceedings of the 6th international conference on Independent Component Analysis and Blind Signal Separation

Distributed high dimensional information theoretical image registration via random projections

Digital Signal Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task. We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.